Ocelot: Not Found (404) Error with Service Discovery - Docker / Consul

Expected Behavior

Reroute to the downstream path.

Actual Behavior

Any request return a Not Found Error(404) with the following message in the VS output window:

warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]  requestId: 0HM2E7R13PE70:00000001, previousRequestId: no previous request id, message: Error Code: ServicesAreEmptyError Message: services were empty for SharepointAPI errors found in ResponderMiddleware. Setting error response for request path:/File, request method: GET

Ocelot.Responder.Middleware.ResponderMiddleware: Warning: requestId: 0HM2E7R13PE70:00000001, previousRequestId: no previous request id, message: Error Code: ServicesAreEmptyError Message: services were empty for SharepointAPI errors found in ResponderMiddleware. Setting error response for request path:/File, request method: GET

Steps to Reproduce the Problem

I created two ASP Net Core Web API projects. The first one implement a API Gateway using Ocelot with Consul and the second one is a simple Web API Service with a endpoint routed to /api/v1/File. Both project are initialized by a docker composer project

Ocelot.json file:

"Routes": [
   {
     "DownstreamPathTemplate": "/api/v1/File",      
     "DownstreamScheme": "http",
     "UpstreamPathTemplate": "/File",
     "UpstreamHttpMethod": [ "Get" ],
     "ServiceName": "SharepointAPI",
     "LoadBalancerOptions": {
       "Type": "LeastConnection"
     }
   }
 ],
 "GlobalConfiguration": {
   "ServiceDiscoveryProvider": {
     "Scheme": "http",
     "Host": "consul",
     "Port": 8500,
     "Type": "PollConsul"
   }
 }

Docker composer file

version: '3.4'

services:
    sharepoint.api:
       image: ${DOCKER_REGISTRY-}orderingapi
       hostname: sharepointapi
       build:
           context: .
           dockerfile: Sharepoint.API/Dockerfile
       ports:
           - "8002:80"

   private.gtw:
       image: ${DOCKER_REGISTRY-}privategtw
       build:
           context: .
           dockerfile: Private.Gtw/Dockerfile
      ports:
           - "7000:80"

   consul:
       image: consul:latest
       command: consul agent -dev -log-level=warn -ui -client=0.0.0.0 -bind='{{ GetPrivateIP }}'        
       hostname: consul
       ports:
           - "8500:8500"

Startup file

        public void ConfigureServices(IServiceCollection services)
        {
            services
                .AddOcelot()
                .AddConsul(); 
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseRouting();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });

            //ocelot
            app.UseOcelot().Wait();
        }

Output from Consul Monitor

/ # consul monitor
2020-09-01T06:54:40.712Z [INFO]  agent: Deregistered service: service=SharepointAPI
2020-09-01T06:54:40.911Z [INFO]  agent: Synced service: service=SharepointAPI

Specifications

The service looks registered in the web client. If I ping beetwen the consul container, the private gateway and the service container they can be reached each other.

Nuget Ocelot Version: 16.0.1
Nuget Ocelot.Provider.Consul Version: 16.0.1
Nuget Consul Version: 1.6.1.1
Platform: Net Core 3.1
Subsystem: Windows 10

How can I solve this problem?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (6 by maintainers)

Commits related to this issue

#1634 #1487 #1329 #1304 #1294 #793 Consul polling of services: enhancements and fix errors (#1670) * fixing some issues in poll consul: - Timer is not thread safe, avoiding usage of it - No Ressour... — committed to ThreeMammals/Ocelot by ggnaegi 9 months ago

Most upvoted comments

@johha86 sure, but then it’s for every request and I thought it would help if we wouldn’t keep retrieving the services on every request. In my case, I can see quite a performance improvement. It’s a bit chatty otherwise, you call _consulServiceDiscoveryProvider.Get which then call consul to get the consul clients information on every request.

So I don’t want to avoid polling per se, I’m just looking for a solution that is not too chatty…

It’s an hybrid implementation then: Per request until services returned, then like a polling, but trying to avoid imho race conditions induced by the timer.

To summarize:

I was like you hoping to have a working polling to reduce calls to consul service
I tried your solution but it didn’t quite work for me, because the polling is started per template route I presume not per service, so I ended up having the same problems as @3dotsDev (my ocelot file is big).
I updated the code to maintain a collection of service discovery providers
I have some services that aren’t reachable when the Gateway is started. If a call is made (could happen), then I will have to wait until the next timer callback. So, I thought maybe I should try per request until the service is returned.
And finally I got cold feet, and I thought: could possibly a thread get _services=null because the services are updated in the timer callback? And then I came up with this “incredible” solution ! 😃

ggnaegi on Sep 15, 2021

Hello @ggnaegi

If you want to “Avoid polling” I think that you only need to setup Ocelot following the instruction: "ServiceDiscoveryProvider": { "Scheme": "https", "Host": "localhost", "Port": 8500, "Type": "Consul" } The type Consul retrieve the services until they are available on every request. You can found more details here. The problem that I faced in this issue is when the Type PollConsul is used in the service discovery configuration

johha86 on Sep 15, 2021

Hi, I had found the reason of the problem and how to fixe it. When you configure Ocelot to use Service Discovery as PollConsul mode, an IServiceDiscoveryProvider is created with a timer in the Constructor. In the callback of the timer, happens the Poll of the available services in Consul into a collection. But the initialization of this timer happens at the first time you call Ocelot. So, the first time you call ocelot, such collection is empty and you receive a Not Found 404. The repository with this code is Archived so I can’t do a PR with the Fix. What I did was do a Fork of the repository ,add the fix and use my forked project instead the Nuget Package. This is how the constructor in my Forked project is:

public PollConsul(int pollingInterval, IOcelotLoggerFactory factory, IServiceDiscoveryProvider consulServiceDiscoveryProvider)
{
   _logger = factory.CreateLogger<PollConsul>();
   _consulServiceDiscoveryProvider = consulServiceDiscoveryProvider;
   _services = new List<Service>();

   //  Do a first Poll before the first Timer callback.
   Poll().Wait();

  _timer = new Timer(async x =>
   {
     if (_polling)
     {
        return;
     }
  
     _polling = true;
     await Poll();
     _polling = false;
   }, null, pollingInterval, pollingInterval);
}

Other option is implement an IHostedService that perform calls to an endpoint at the beginning of the execution of the application.

I will keep this issue open because the original code doesn’t have a solution yet. Maybe I could maintain the original repository.

johha86 on Aug 14, 2021

Have you tried to repeat the request? I also receive the 404 status code the first time when I use PollConsul type, but subsequent requests are handled well. And it is not dependent on the use of containers.

jlukawska on Oct 22, 2020