Observation's Shape

[catarinam93]

I have trained a model using LidarObservation as following:

model = PPO('MlpPolicy', env,
            policy_kwargs=dict(net_arch=[256, 256]),
            learning_rate=5e-4,
            n_steps=2048,
            batch_size=64,
            n_epochs=10,
            gamma=0.8,
            gae_lambda=0.95,
            clip_range=0.2,
            verbose=1,
            tensorboard_log="highway_ppo_lidar_40/")
timesteps = 50000
model.learn(total_timesteps=timesteps)
model.save("highway_ppo_lidar_40/model")

By default: print(env.observation_space) outputs: Box(-inf, inf, (5, 5), float32). But, only by doing env.reset() it becomes Observation Shape after Reset: (16, 2).

I was trying to test the model by loading it model = Algorithm.load(model_path) and running a few episodes but when obs, info = env.reset() is done, the shape changes to (16,2). Therefore, when doing model.predict() it gives me the following error: Error: Unexpected observation shape (16, 2) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.

for episode in range(n_episodes):
        done = truncated = False
        obs, info = env.reset()
        episode_reward = 0
        time_steps = 0
        while not (done or truncated):
            action, _states = model.predict(obs, deterministic=True) # ERROR HERE
            new_obs, reward, done, truncated, info = env.step(action)
            episode_reward += reward
            time_steps += 1

The same happens for TimeToCollision which is Box(-inf, inf, (5, 5), float32) and becomes (3, 3, horizon)

Does anyone know what's happening?

[eleurent]

• (5, 5) is the (default) shape associated with the (default) Kinematics observation: it describes the 5 closest vehicles, with 5 features each (x, y, vx, vy, presence)

• (16. 2) is the (default) shape associated with the Lidar observation: 16 angular directions x 2 features (distance and radial speed).

You are probably doing something like this.

# 1. here the env uses the default observation type, so shape is (5, 5)
env = gymnasium.make('env-id')

# 2. changing the observation type to lidar observation --> will change it to (16, 2) at the next reset
env.config['observation'] = {'type': 'LidarObservation'}

# 3. initialise model --> shape is still (5, 5) so policy params are expecting a (5, 5) input
model = PPO(env, ...)

# 4. will trigger an env.reset() which will change the shape to (16, 2) and make it 
model.learn()

The problem is that changing the configuration is "lazy", it only has an effect when the environment is reset (e.g. you can't change the number of lanes in the middle of a simulation). So while you've change the configuration at step 2., it's not effective immediately and the observation shape will remain (5, 5) until the env is reset at 4., but by then the model will have been initialised at 3. expecting a (5, 5) shape.

The solution is to do something like

# here the env uses the default observation type, so shape is (5, 5)
env = gymnasium.make('env-id')

# changing the observation type to lidar observation --> will change it to (16, 2) at the next reset
env.config['observation'] = {'type': 'LidarObservation'}

# new observation space is taken into account -> shape is now (16, 2)
env.reset()

# initialise model --> policy params will expect a (16, 2) shape
model = PPO(env, ...)

# things are fine and consistent
model.learn()

Or even better, pass the desired config to the first initialisation:

# initialise env directly with desired config, so shape is (16, 2)
env = gymnasium.make('env-id', config={'type': 'LidarObservation'})

# initialise model --> policy params will expect a (16, 2) shape
model = PPO(env, ...)

# things are fine and consistent
model.learn()

And the same applies to testing the trained model, the environment has to be properly initialised/reset before.

[catarinam93]

I have done what you first suggest, do env.reset() before training (it doesn't even let you train without doing it). The problem is afterwards when I try to run a few episodes.

First I loaded the model and initialized the environment, then I runned the aforementioned cycle:

for episode in range(n_episodes):
        done = truncated = False
        obs, info = env.reset()
        episode_reward = 0
        time_steps = 0
        while not (done or truncated):
            action, _states = model.predict(obs, deterministic=True) # ERROR HERE
            new_obs, reward, done, truncated, info = env.step(action)
            episode_reward += reward
            time_steps += 1

But on the predict step, due to the obs, info = env.reset() ,it gives an error :
For TimeToCollision: Error: Unexpected observation shape (3, 3, 20) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.
For LidarObservation: Error: Unexpected observation shape (16, 2) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.

The data file of the models contains:
For TimeToCollision:

"observation_space": {
        ":type:": "<class 'gymnasium.spaces.box.Box'>",
        ":serialized:": (I deleted this here so it wouldn’t be huge :))
        "dtype": "float32",
        "bounded_below": "[[[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]\n\n [[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]\n\n [[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]]",
        "bounded_above": "[[[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]\n\n [[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]\n\n [[ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]\n  [ True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True  True  True  True\n    True  True  True  True  True  True  True  True]]]",
        "_shape": [
            3,
            3,
            30
        ],
        "low": "[[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]]\n\n [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]]\n\n [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]\n  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n   0. 0. 0. 0. 0. 0. 0.]]]",
        "high": "[[[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]]\n\n [[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]]\n\n [[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]\n  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n   1. 1. 1. 1. 1. 1. 1.]]]",
        "low_repr": "0.0",
        "high_repr": "1.0",
        "_np_random": null
    },

For LidarObservation:

"observation_space": {
        ":type:": "<class 'gymnasium.spaces.box.Box'>",
        ":serialized:": "gAWVyQIAAAAAAACMFGd5bW5hc2l1bS5zcGFjZXMuYm94lIwDQm94lJOUKYGUfZQojAVkdHlwZZSMBW51bXB5lIwFZHR5cGWUk5SMAmY0lImIh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMDWJvdW5kZWRfYmVsb3eUjBJudW1weS5jb3JlLm51bWVyaWOUjAtfZnJvbWJ1ZmZlcpSTlCiWIAAAAAAAAAABAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAZRoCIwCYjGUiYiHlFKUKEsDjAF8lE5OTkr/////Sv////9LAHSUYksQSwKGlIwBQ5R0lFKUjA1ib3VuZGVkX2Fib3ZllGgRKJYgAAAAAAAAAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBlGgVSxBLAoaUaBl0lFKUjAZfc2hhcGWUSxBLAoaUjANsb3eUaBEoloAAAAAAAAAAAACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL8AAIC/AACAvwAAgL+UaAtLEEsChpRoGXSUUpSMBGhpZ2iUaBEoloAAAAAAAAAAAACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD8AAIA/AACAPwAAgD+UaAtLEEsChpRoGXSUUpSMCGxvd19yZXBylIwELTEuMJSMCWhpZ2hfcmVwcpSMAzEuMJSMCl9ucF9yYW5kb22UTnViLg==",
        "dtype": "float32",
        "bounded_below": "[[ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]]",
        "bounded_above": "[[ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]\n [ True  True]]",
        "_shape": [
            16,
            2
        ],
        "low": "[[-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]\n [-1. -1.]]",
        "high": "[[1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]\n [1. 1.]]",
        "low_repr": "-1.0",
        "high_repr": "1.0",
        "_np_random": null
    },

Why does it expect the kinematics's shape??

[eleurent]

You're right, it looks like training worked as intended and the observation shapes expected by the policies are correct.
I'm not sure why the observation are misshaped at inference time, can you share your complete launch script and also the full stack trace?

[catarinam93]

# Function to load the environment configuration
def detect_algorithm(folder, env_config_mapping):
    algorithm_name = env_config_mapping.get(folder, {}).get("model", None)
    
    algorithm_mapping = {
        "PPO": PPO,
        "A2C": A2C,
        "DQN": DQN
    }
    
    return algorithm_mapping.get(algorithm_name)

# Function to create the environment
def create_environment(folder, env_config_mapping, observation_type):
    env_config = env_config_mapping.get(folder, {})
    env = gym.make("highway-v0")
    env.unwrapped.configure(env_config)
    return env

def extract_features_from_model(model, env, n_episodes=100):
    # Initialize the performance data list
    performance_data = []

    for episode in range(n_episodes):
        done = truncated = False
        obs, info = env.reset()
        episode_reward = 0
        time_steps = 0

        while not (done or truncated):
            print("Obs Shape - extract function: ", obs.shape)
            action, _states = model.predict(obs, deterministic=True) # ERROR HERE
            new_obs, reward, done, truncated, info = env.step(action)
            episode_reward += reward
            time_steps += 1
    
            # Append the performance data
            performance_data.append({
                'Reward': reward,
                'Step': time_steps,
                'Speed': info.get('speed', np.nan),
                'Crashed': info.get('crashed', False),
                'Collision Reward': info.get('rewards', {}).get('collision_reward', 0),
                'Right Lane Reward': info.get('rewards', {}).get('right_lane_reward', 0),
                'High Speed Reward': info.get('rewards', {}).get('high_speed_reward', 0),
                'On Road Reward': info.get('rewards', {}).get('on_road_reward', 0),
            })

            obs = new_obs

    # Create a DataFrame with the performance data
    performance_df = pd.DataFrame(performance_data)

    # Calculate the metrics
    average_reward = performance_df['Reward'].mean()
    reward_stddev = performance_df['Reward'].std()
    observation_variance = performance_df['Speed'].var()
    action_variance = performance_df['Reward'].var()
    total_transitions = len(performance_df)
    crash_rate = performance_df['Crashed'].mean()
    average_collision_reward = performance_df['Collision Reward'].mean()
    average_right_lane_reward = performance_df['Right Lane Reward'].mean()
    average_high_speed_reward = performance_df['High Speed Reward'].mean()
    average_on_road_reward = performance_df['On Road Reward'].mean()
    average_episode_length = performance_df['Step'].max()
    reward_sparsity = (performance_df['Reward'] != 0).mean() 
    transition_rewards_variance = performance_df['Reward'].var()

    # Obtain the observation type
    observation_type = env.unwrapped.config.get('observation', {}).get('type', 'Unknown')

    # Verify if the observation type is LidarObservation or TimeToCollision
    max_range = None
    horizon = None
    if observation_type == "LidarObservation":
        max_range = env.unwrapped.config.get('observation', {}).get('max_range', 'N/A')
    elif observation_type == "TimeToCollision":
        horizon = env.unwrapped.config.get('observation', {}).get('horizon', 'N/A')

    # Obtain the initial lane id, ego spacing and algorithm name
    initial_lane_id = env.unwrapped.config.get('initial_lane_id', 'N/A')
    ego_spacing = env.unwrapped.config.get('ego_spacing', 'N/A')
    algorithm_name = model.__class__.__name__

    # Dictionary with the environment configuration
    env_config = {
        'Duration': env.unwrapped.config.get('duration', 'N/A'),
        'Number of Lanes': env.unwrapped.config.get('lanes_count', 'N/A'),
        'Number of Vehicles': env.unwrapped.config.get('vehicles_count', 'N/A'),
        'Vehicles Density': env.unwrapped.config.get('vehicles_density', 'N/A'),
        'Reward Speed Range': str(env.unwrapped.config.get('reward_speed_range', 'N/A')),
        'Scaling Factor': env.unwrapped.config.get('scaling', 'N/A'),
        'Algorithm': algorithm_name,
        'Average Reward': average_reward,
        'Reward Standard Deviation': reward_stddev,
        'Observation Variance': observation_variance,
        'Action Variance': action_variance,
        'Number of Transitions Observed': total_transitions,
        'Crash Rate': crash_rate,
        'Average Collision Reward': average_collision_reward,
        'Average Right Lane Reward': average_right_lane_reward,
        'Average High Speed Reward': average_high_speed_reward,
        'Average On Road Reward': average_on_road_reward,
        'Average Episode Length': average_episode_length,
        'Reward Sparsity': reward_sparsity, 
        'Transition Rewards Variance': transition_rewards_variance,
        'Observation Type': observation_type, 
        'Maximum Range (Lidar)': max_range, 
        'Horizon (TimeToCollision)': horizon, 
        'Initial Lane Id': initial_lane_id,
        'Ego Spacing': ego_spacing, 
    }

    # Create a DataFrame with the environment configuration
    summary_df = pd.DataFrame([env_config])

    return {
        'summary_df': summary_df,
    }

# Base Path to the models
base_path = r'C:\Users\Caty\Documents'

# Load the environment configuration mapping
try:
    with open('environment_config.json', 'r') as f:
        env_config_mapping = json.load(f)
except FileNotFoundError:
    print("File 'environment_config.json' not found.")
    env_config_mapping = {}

# List of all models data
all_models_data = []

# List of all summaries
all_summaries = []

# File containing the paths of the models to be ignored - the ones containin LidarObservation or TimeToCollision
ignored_models_file = 'ignored_models.txt'

# Load the paths of the models to be ignored
try:
    with open(ignored_models_file, 'r') as f:
        ignored_model_paths = f.read().splitlines()
except FileNotFoundError:
    print(f"File '{ignored_models_file}' not found.")
    ignored_model_paths = []

# Loop through all the models
for model_path in ignored_model_paths:
    try:
        folder = os.path.basename(os.path.dirname(model_path))

        observation_type = env_config_mapping.get(folder, {}).get('observation', {}).get('type', 'Unknown')

        Algorithm = detect_algorithm(folder, env_config_mapping)
        
        if Algorithm is None:
            raise ValueError(f"Algorithm not detected for: {folder}")
        model = Algorithm.load(model_path)

        env = create_environment(folder, env_config_mapping, observation_type)

        model.set_env(env)

        extracted_data = extract_features_from_model(model, env, n_episodes=100)
        print(extracted_data['summary_df'])

        if not extracted_data['summary_df'].empty:
            all_summaries.append(extracted_data['summary_df'])

    except Exception as e:
        print(f"Error loading model {model_path}: {e}")

# Combine all summaries into a single DataFrame
if all_summaries:
    summary_df_combined = pd.concat(all_summaries, ignore_index=True)
    summary_df_combined.to_csv('summary_df_combined.csv', index=True)

Output:

Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
Obs Shape - extract function:  (3, 3, 20)
Error loading model C:\Users\Caty\Documents\highway_comb_env_1\model.zip: Error: Unexpected observation shape (3, 3, 20) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.
Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
C:\Users\Caty\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\highway_env\utils.py:399: RuntimeWarning: divide by zero encountered in scalar divide
  interval_1 = [(a - r) @ u / rqu, (b - r) @ u / rqu]
C:\Users\Caty\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gymnasium\utils\passive_env_checker.py:135: UserWarning: �[33mWARN: The obs returned by the `reset()` method was expecting numpy array dtype to be float32, actual type: float64�[0m
  logger.warn(
C:\Users\Caty\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gymnasium\utils\passive_env_checker.py:159: UserWarning: �[33mWARN: The obs returned by the `reset()` method is not within the observation space.�[0m
  logger.warn(f"{pre} is not within the observation space.")
Obs Shape - extract function:  (16, 2)
Error loading model C:\Users\Caty\Documents\highway_comb_env_10\model.zip: Error: Unexpected observation shape (16, 2) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.
Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
Obs Shape - extract function:  (3, 3, 30)
Error loading model C:\Users\Caty\Documents\highway_comb_env_11\model.zip: Error: Unexpected observation shape (3, 3, 30) for Box environment, please use (5, 5) or (n_env, 5, 5) for the observation shape.

...