# Pastebin m4PJgmxP $ sudo -E kubectl logs -n openstack --follow neutron-l3-agent-default-2kmd8 Error from server (NotFound): pods "neutron-l3-agent-default-2kmd8" not found (reverse-i-search)`-n ': sudo -E kubectl logs ^C openstack --follow neutron-l3-agent-default-2kmd8 ks3019@cab23-r720-11:~$ watch 'sudo -E kubectl get pods -n openstack -o wide |grep -v Comple |grep l3' ks3019@cab23-r720-11:~$ sudo -E kubectl logs -n openstack --follow neutron-l3-agent-default-gf8qq + exec neutron-l3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini --config-file /etc/neutron/metadata_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. 2019-08-23 02:25:29.229 72 INFO neutron.common.config [-] Logging enabled! 2019-08-23 02:25:29.229 72 INFO neutron.common.config [-] Logging enabled! 2019-08-23 02:25:29.233 72 INFO neutron.common.config [-] /var/lib/openstack/bin/neutron-l3-agent version 10.0.8.dev145 2019-08-23 02:25:29.233 72 INFO neutron.common.config [-] /var/lib/openstack/bin/neutron-l3-agent version 10.0.8.dev145 2019-08-23 02:25:29.399 72 INFO neutron.agent.agent_extensions_manager [req-01c25d18-6b2e-43fd-8565-33d7a7ae7b29 - - - - -] Loaded agent extensions: [] 2019-08-23 02:25:29.399 72 INFO neutron.agent.agent_extensions_manager [req-01c25d18-6b2e-43fd-8565-33d7a7ae7b29 - - - - -] Loaded agent extensions: [] /var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:200: FutureWarning: The access_policy argument is changing its default value to in version '?', please update the code to explicitly set None as the value: access_policy defaults to LegacyRPCAccessPolicy which exposes private methods. Explicitly set access_policy to DefaultRPCAccessPolicy or ExplicitRPCAccessPolicy. access_policy) 2019-08-23 02:25:29.490 72 INFO neutron.agent.l3.agent [-] L3 agent started 2019-08-23 02:25:29.490 72 INFO neutron.agent.l3.agent [-] L3 agent started 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:25:58.401 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:26:13.366 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:26:28.357 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:26:43.489 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:26:56.392 72 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connecting... 2019-08-23 02:26:56.392 72 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connecting... 2019-08-23 02:26:56.393 72 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connected 2019-08-23 02:26:56.393 72 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connected 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:26:58.463 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:27:01.651 72 INFO neutron.agent.l3.ha [-] Router 780ea599-d192-4cdd-9f9e-52476ee8d201 transitioned to backup 2019-08-23 02:27:01.651 72 INFO neutron.agent.l3.ha [-] Router 780ea599-d192-4cdd-9f9e-52476ee8d201 transitioned to backup 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors 2019-08-23 02:27:13.657 72 ERROR oslo_messaging.rpc.server 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server [-] Exception during message handling 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 227, in dispatch 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server raise NoSuchMethod(method) 2019-08-23 02:27:28.730 72 ERROR oslo_messaging.rpc.server NoSuchMethod: Endpoint does not support RPC method pod_health_probe_method_ignore_errors