Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I have GPU GeForce GTX 1050 and 32GB RAM but still issuse of OOM #55

Open
haideralifaridkhan opened this issue Jan 12, 2021 · 2 comments

Comments

@haideralifaridkhan
Copy link

199/200 [============================>.] - ETA: 0s - loss: 17700.6033 - PSNR: 10.1373Traceback (most recent call last):
File "train.py", line 112, in
main()
File "train.py", line 106, in main
callbacks=callbacks)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1861, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1141, in fit
return_dict=True)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1389, in evaluate
tmp_logs = self.test_function(iterator)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 828, in call
result = self._call(*args, **kwds)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 871, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 726, in _initialize
*args, **kwds))
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 2969, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3206, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 634, in wrapped_fn
out = weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 977, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1233 test_function  *
    return step_function(self, iterator)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1224 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib64/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1259 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2730 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:3417 _call_for_each_replica
    return fn(*args, **kwargs)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1217 run_step  **
    outputs = model.test_step(data)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1183 test_step
    y_pred = self(x, training=False)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:998 __call__
    input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/input_spec.py:207 assert_input_compatibility
    ' input tensors. Inputs received: ' + str(inputs))

ValueError: Layer model expects 1 input(s), but it received 2 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, None, None, None) dtype=uint8>, <tf.Tensor 'IteratorGetNext:1' shape=(None, None, None, None) dtype=uint8>]

2165-01-12 17:11:32.737853: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]

@haideralifaridkhan
Copy link
Author

How much minimum specification computer required to performed this code?

@Andyshen555
Copy link

This is not a problem of OOM. You have to change the code in generator py to make the list into a tuple, since now validation set only accepts tuple instead of list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants