com.google.api.control.ConfigFilter doFilter: Rejecting this API request due to config loading error

[j-soliduslink]

Our API servlet occasionally throws 500 because of
com.google.api.control.ConfigFilter doFilter: Rejecting this API request due to config loading error

tracing the root cause at the instance start phase and got
com.google.api.control.ConfigFilter init: Failed to load service: %s (ConfigFilter.java:93)
com.google.api.config.ServiceConfigException: java.net.SocketTimeoutException: connect timed out
at com.google.api.config.ServiceConfigSupplier.fetch(ServiceConfigSupplier.java:155)
at com.google.api.config.ServiceConfigSupplier.get(ServiceConfigSupplier.java:131)
at com.google.api.config.ServiceConfigSupplier.get(ServiceConfigSupplier.java:50)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:240)
at com.google.api.config.ServiceConfigFetcher.fetch(ServiceConfigFetcher.java:41)
at com.google.api.control.ServiceManagementConfigFilter$1.load(ServiceManagementConfigFilter.java:33)
at com.google.api.control.ConfigFilter.init(ConfigFilter.java:89)
at com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:112)
at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:99)
at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:220)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1406)
at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.startWebapp(AppEngineWebAppContext.java:159)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1368)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:522)
at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doStart(AppEngineWebAppContext.java:116)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:244)
at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:182)
at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:109)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:693)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:655)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:625)
at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:819)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:670)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:104)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.config.ServiceConfigSupplier.fetch(ServiceConfigSupplier.java:153)
... 29 more

the timeout exception cause the API instance in a bad state and failed to serve any requests afterwards.

[cybersokari]

I'm having this same issue too

[v4ires]

I have the same issue.

[JoePacman]

We have the same issue

[siliconsage]

I am also facing the issue. But when I restart the server it works fine.