Lifelong learning supporting non-structure #352

jaypume · 2022-08-22T01:18:50Z

No description provided.

Signed-off-by: luosiqi <[email protected]>

Sedna lifelong learning supports unstructured data based on semantic segmentation example

Signed-off-by: luosiqi <[email protected]>

Code check and base model improvement of unstructured lifelong learning framework

JimmyYang20 · 2022-08-22T01:22:03Z

@luosiqi Delete code files irrelevant to the scenarios in example folder.

JimmyYang20 · 2022-08-22T01:25:00Z

examples/lifelong_learning/RFNet/basemodel.py

+    return CPA
+
+
+if __name__ == '__main__':


@luosiqi put the test code elsewhere, e.g. ./test/test_basemodel.py

JimmyYang20 · 2022-08-22T01:31:39Z

examples/lifelong_learning/RFNet/basemodel.py

+
+
+def train_args():
+    parser = argparse.ArgumentParser(description="PyTorch RFNet Training")


@luosiqi
The command-line parsing module argparse should not be used, because it dose not use in this scense. It's easy to misunderstand.

@luosiqi suggest that:

Class TrainArgs: def __init__(self, **kwargs): self.depth = kwargs.get('depth', False) self.dateaset = Context.get_parameters('dataset', 'cityscapes') ``

JimmyYang20 · 2022-08-22T01:54:33Z

examples/lifelong_learning/RFNet/basemodel.py

+                    'best_pred': self.trainer.best_pred,
+                }, is_best)
+
+            # if not self.trainer.args.no_val and \


@luosiqi delete comment code

JimmyYang20 · 2022-08-22T01:56:55Z

examples/lifelong_learning/RFNet/basemodel.py

+    return args
+
+
+def accuracy(y_true, y_pred, **kwargs):


@luosiqi the ./accuracy.py has this func accuracy in the project, so you can import it.

JimmyYang20 · 2022-08-22T01:58:18Z

examples/lifelong_learning/RFNet/basemodel.py

+from dataloaders import make_data_loader
+from dataloaders import custom_transforms as tr
+
+def preprocess(image_urls):


this func may be the class(Model)‘s private func

JoeyHwong-gk · 2022-08-22T02:04:57Z

examples/lifelong_learning/RFNet/accuracy.py

+from utils.metrics import Evaluator
+from tqdm import tqdm
+from dataloaders import make_data_loader
+from sedna.common.class_factory import ClassType, ClassFactory


note the order of import

JoeyHwong-gk · 2022-08-22T02:05:35Z

examples/lifelong_learning/RFNet/accuracy.py

@@ -0,0 +1,38 @@
+from basemodel import val_args


The import of the relative path should be adjusted.

JoeyHwong-gk · 2022-08-22T02:06:25Z

examples/lifelong_learning/RFNet/accuracy.py

+__all__ = ('accuracy')
+
+@ClassFactory.register(ClassType.GENERAL)
+def accuracy(y_true, y_pred, **kwargs):


Common keyword. Use alias while register.

JoeyHwong-gk · 2022-08-22T02:06:38Z

examples/lifelong_learning/RFNet/accuracy.py

+    _, _, test_loader, num_class = make_data_loader(args, test_data=y_true)
+    evaluator = Evaluator(num_class)
+
+    tbar = tqdm(test_loader, desc='\r')


JoeyHwong-gk · 2022-08-22T02:07:18Z

examples/lifelong_learning/RFNet/accuracy.py

+        if args.cuda:
+            image, target = image.cuda(), target.cuda()
+            if args.depth:
+                depth = depth.cuda()


Check whether the device supports GPU.

JoeyHwong-gk · 2022-08-22T02:10:54Z

examples/lifelong_learning/RFNet/basemodel.py

+            'cityrand',
+            'target',
+            'xrlab',
+            'e1',


what's the meanning of xrlab and e1

JoeyHwong-gk · 2022-08-22T02:12:06Z

examples/lifelong_learning/RFNet/basemodel.py

+
+    if args.checkname is None:
+        args.checkname = 'RFNet'
+    print(args)


relace print by using logger

JoeyHwong-gk · 2022-08-22T02:12:40Z

examples/lifelong_learning/RFNet/basemodel.py

+        choices=[
+            'citylostfound',
+            'cityscapes',
+            'xrlab',


Seems to be inconsistent with the training

JoeyHwong-gk · 2022-08-22T02:14:45Z

examples/lifelong_learning/RFNet/dataloaders/datasets/cityrand.py

+from dataloaders import custom_transforms as tr
+
+class CityscapesSegmentation(data.Dataset):
+    NUM_CLASSES = 19


magic number

JoeyHwong-gk · 2022-08-22T02:15:27Z

examples/lifelong_learning/RFNet/dataloaders/datasets/cityscapes.py

+    def __init__(self, args, root=Path.db_root_dir('cityscapes'), data=None, split="train"):
+
+        # self.root = root
+        self.root = "/home/lsq/Dataset/"


JoeyHwong-gk · 2022-08-22T02:18:36Z

examples/lifelong_learning/RFNet/models/rfnet.py

@@ -0,0 +1,27 @@
+import torch.nn as nn
+from itertools import chain # 串联多个迭代对象


replace with english will be more general

JoeyHwong-gk · 2022-08-22T02:19:51Z

examples/lifelong_learning/RFNet/models/resnet/resnet_single_scale_single_attention_unseen.py

+    Args:
+        pretrained (bool): If True, returns a model pre-trained on ImageNet
+    """
+    model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)


[2, 2, 2, 2] Hyperparameters are restricted.

JoeyHwong-gk · 2022-08-22T02:21:21Z

examples/lifelong_learning/RFNet/models/replicate.py

@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+# File   : replicate.py
+# Author : Jiayuan Mao


Be aware of the use of other people's code under community constraints

JoeyHwong-gk · 2022-08-22T02:22:49Z

examples/lifelong_learning/RFNet/dataloaders/utils.py

+        label_colours = get_cityscapes_labels()
+    elif dataset == 'target':
+        n_classes = 24
+        label_colours = get_cityscapes_labels()


switch Statements

kubeedge-bot · 2022-08-22T02:22:59Z

@JoeyHwong-gk: changing LGTM is restricted to collaborators

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

JoeyHwong-gk · 2022-08-22T02:23:53Z

Please add kubeedge copyright at the top

JoeyHwong-gk · 2022-08-22T12:12:00Z

lib/sedna/algorithms/knowledge_management/edge_knowledge_management.py

+        self.extractor_key = KBResourceConstant.EXTRACTOR.value
+
+        ModelLoadingThread(self, self.task_index).start()
+


Lots of repetitive code, please move up.

JoeyHwong-gk · 2022-08-22T12:12:40Z

lib/sedna/algorithms/knowledge_management/edge_knowledge_management.py

+            try:
+                task_index = FileOps.load(task_index_url)
+            except Exception as err:
+                self.log.error(f"{err}")


proposed merge

JoeyHwong-gk · 2022-08-22T12:18:45Z

lib/sedna/common/class_factory.py

@@ -37,6 +37,11 @@ class ClassType:
    DATASET = 'data_process'
    CALLBACK = 'post_process_callback'

+    # TODO


what the todo tags for?

JoeyHwong-gk · 2022-08-22T12:20:57Z

lib/sedna/algorithms/unseen_task_processing/unseen_task_allocation/unseen_task_allocation.py

+
+    def __init__(self, task_extractor, **kwargs):
+        self.task_extractor = task_extractor
+        self.log = LOGGER


what's the reasons to define self.log ?

JoeyHwong-gk · 2022-08-22T12:22:22Z

lib/sedna/algorithms/seen_task_learning/task_definition/task_definition.py

+        for i in range(self.n_class):
+            # sample = BaseDataSource()
+            # sample.x = samples.x[i * partition_length: (i + 1) * partition_length]
+            # sample.y = samples.y[i * partition_length: (i + 1) * partition_length]


JimmyYang20 · 2022-08-23T11:54:09Z

@luosiqi
Note that each line code cannot contain more than 80 characters. Otherwise, the CI check fails.

JimmyYang20 · 2022-08-23T11:58:03Z

examples/lifelong_learning/RFNet/basemodel.py

+        self.val_args.label_save_path = os.path.join(label_save_dir, "label")
+        self.val_args.save_predicted_image = kwargs.get(
+            "save_predicted_image", "true").lower()
+        self.validator = Validator(self.val_args)


It is not recommended that self.validator = Validator(self.val_args) be placed in the initialization phase.

JimmyYang20 · 2022-08-23T12:00:39Z

examples/lifelong_learning/RFNet/dataloaders/datasets/e1.py

+from dataloaders import custom_transforms as tr
+
+class CityscapesSegmentation(data.Dataset):
+    NUM_CLASSES = 24


magic number

JimmyYang20 · 2022-08-23T12:03:23Z

lib/sedna/datasources/__init__.py

+    txt file which contain image list parser
+    """
+
+    def __init__(self, data_type, func=None):


Use func to handle it!

# func may use this # func = _data_feature_process def _data_feature_process(line: str): res = line.strip().split() return res[:-1], res[-1]

JimmyYang20 · 2022-08-23T12:05:27Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+                KBResourceConstant.EDGE_KB_DIR.value),
+            task_index=KBResourceConstant.KB_INDEX_NAME.value)
+
+        self.cloud_knowledge_management = CloudKnowledgeManagement(


Don't put it in the this func.
you can put it in "train func" and "eval func"

JimmyYang20 · 2022-08-23T12:05:45Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+        self.cloud_knowledge_management = CloudKnowledgeManagement(
+            config, estimator=e)
+
+        self.edge_knowledge_management = EdgeKnowledgeManagement(


Don't put it in the this func.
you can put it in "infer func"

JimmyYang20 · 2022-08-23T12:07:13Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+            self.cloud_knowledge_management,
+            self.edge_knowledge_management,
+            unseen_task_allocation)
+
        task_index = FileOps.join_path(config['output_url'],


CloudKnowledgeManagement also has this command, delete it?

JimmyYang20 · 2022-08-23T12:07:50Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+
+        seen_samples, unseen_samples = unseen_sample_re_recognition(train_data)
+
+        # TODO: retrain temporarily


Delete the omment

JimmyYang20 · 2022-08-23T12:12:05Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

            relpath=self.config.data_path_prefix)
        self.report_task_info(
            None, K8sResourceKindStatus.COMPLETED.value, task_info_res)
        self.log.info(f"Lifelong learning Train task Finished, "
-                      f"KB idnex save in {self.config.task_index}")
+                      f"KB index save in {task_index}")
        return callback_func(self.estimator, res) if callback_func else res

    def update(self, train_data, valid_data=None, post_process=None, **kwargs):


combine this funcupdate and train into an external func, e,g,:

def train(self): if not has_completed_initial_training: return self._initial_train() return self._update(self)

Combine A and B into an external function.

JimmyYang20 · 2022-08-23T12:16:03Z

examples/lifelong_learning/RFNet/sedna_train.py

+    train_data = IndexDataParse(data_type="train", func=_load_txt_dataset)
+    train_data.parse(train_dataset_url, use_raw=False)
+
+    is_completed_initilization = str(Context.get_parameters("HAS_COMPLETED_INITIAL_TRAINING", "false")).lower()


Put this judgment in the sedna lib

Only one train interface is exposed to users.

JimmyYang20 · 2022-08-23T12:32:30Z

lib/sedna/algorithms/unseen_task_processing/unseen_task_processing.py

+        self.estimator = set_backend(estimator=estimator, config=config)
+        self.cloud_knowledge_management = cloud_knowledge_management
+        self.edge_knowledge_management = edge_knowledge_management
+


put parameters（cloud_knowledge_management and edge_knowledge_management）to other funs instead of initial func.

JimmyYang20 · 2022-08-23T16:36:26Z

lib/sedna/algorithms/seen_task_learning/seen_task_learning.py

+
+        feedback = {}
+        for i, task in enumerate(task_groups):
+            LOGGER.info(f"MTL Train start {i} : {task.entry}")


task.samples may be [ ]

xiaochanwang · 2022-09-01T07:02:24Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+        self.task_update_decision = task_update_decision or {
+            "method": "UpdateStrategyDefault"
+        }
+        self.task_update_decision_param = e._parse_param(


if task_update_decision is a callable module instance, then there's no need to set its param by _parse_param.

xiaochanwang · 2022-09-01T07:04:56Z

lib/sedna/core/lifelong_learning/lifelong_learning.py

+        seen_samples.y = np.concatenate(
+            (seen_samples.y, unseen_samples.y), axis=0)
+
+        task_update_decision = ClassFactory.get_cls(


if task_update_decision is callable, then skip ClassFactory.get_cls method and set task index instead.

xiaochanwang · 2022-09-01T07:10:39Z

lib/sedna/algorithms/knowledge_management/task_update_decision/task_update_decision.py

+
+
+@ClassFactory.register(ClassType.KM)
+class UpdateStrategyDefault:


Add set method to set task index.

Signed-off-by: JimmyYang20 <[email protected]>

Add pylint in ci

kubeedge-bot · 2022-09-08T04:11:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubeedge-bot · 2022-11-02T06:49:14Z

@jaypume: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

luosiqi and others added 5 commits August 12, 2022 22:15

Add lifelong learning example: semantic segmentation

98a4047

Signed-off-by: luosiqi <[email protected]>

Sedna integrates unstructured data in lib

e5d2291

Signed-off-by: luosiqi <[email protected]>

Merge pull request #349 from luosiqi/dev-lifelong-n

53e5ae4

Sedna lifelong learning supports unstructured data based on semantic segmentation example

Code check and base model improvement

88827ef

Signed-off-by: luosiqi <[email protected]>

Merge pull request #350 from luosiqi/dev-lifelong-n

3ed85a0

Code check and base model improvement of unstructured lifelong learning framework

kubeedge-bot requested review from JimmyYang20 and TymonXie August 22, 2022 01:18

kubeedge-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 22, 2022

JimmyYang20 reviewed Aug 22, 2022

View reviewed changes

JoeyHwong-gk reviewed Aug 22, 2022

View reviewed changes

JoeyHwong-gk suggested changes Aug 22, 2022

View reviewed changes

JoeyHwong-gk reviewed Aug 22, 2022

View reviewed changes

JimmyYang20 reviewed Aug 23, 2022

View reviewed changes

xiaochanwang reviewed Sep 1, 2022

View reviewed changes

JimmyYang20 and others added 2 commits September 8, 2022 10:59

Add pylint in ci

572f4e3

Signed-off-by: JimmyYang20 <[email protected]>

Merge pull request #361 from JimmyYang20/lifelong-ci

e12ccac

Add pylint in ci

kubeedge-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 2, 2022



		def train_args():
		parser = argparse.ArgumentParser(description="PyTorch RFNet Training")

		@@ -0,0 +1,27 @@
		import torch.nn as nn
		from itertools import chain # 串联多个迭代对象

		self.extractor_key = KBResourceConstant.EXTRACTOR.value

		ModelLoadingThread(self, self.task_index).start()


		seen_samples, unseen_samples = unseen_sample_re_recognition(train_data)

		# TODO: retrain temporarily



		@ClassFactory.register(ClassType.KM)
		class UpdateStrategyDefault:

Lifelong learning supporting non-structure #352

Are you sure you want to change the base?

Lifelong learning supporting non-structure #352

Conversation

jaypume commented Aug 22, 2022

JimmyYang20 commented Aug 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kubeedge-bot commented Aug 22, 2022

JoeyHwong-gk commented Aug 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JimmyYang20 commented Aug 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kubeedge-bot commented Sep 8, 2022

kubeedge-bot commented Nov 2, 2022