[BUG] FQE Loading .d3 errors

[jdesman1]

Describe the bug
Perhaps related to #381, I also encounter the bug of loading .d3 checkpoints from FQE. In particular, when using load_learnable, it raises NotImplementedError from

d3rlpy/d3rlpy/ope/fqe.py

Lines 74 to 77 in 3433de5

	def create(self, device: DeviceArg = False) -> "_FQEBase":
	raise NotImplementedError(
	"Config object must be directly given to constructor"
	)

I have also tried initializing fqe along the lines of:

fqe = FQE(algo=model, config=....)
fqe.build_with_dataset(dataset)
fqe.load_model("mymodel.d3")

which yields:

"..../serialization.py", line 1035, in _legacy_load
    raise RuntimeError("Invalid magic number; corrupt file?")
RuntimeError: Invalid magic number; corrupt file?

which is odd, since I save and load .d3 files for other d3rlpy models all the time without an issue. Im happy to run a modified load_learnable if such a workaround can be made

Is there an intended workaround for this?

[takuseno]

@jdesman1 Hi, thanks for the issue. Currently, FQE does not support .d3 format loading. The workaround would be:

# During training
fqe.save_model("fqe.pt")  # save as PyTorch model

# When loading
fqe.load_model("fqe.pt")

This .pt format model is not saved during training by default. If you need to save all checkpoints during training, you can use callback option.

def callback(algo, epoch, total_step):
    algo.save_model(f"{epoch}_model.pt")

fqe.fit(..., epoch_callback=callback)