No healthy upstream ошибка как исправить

This is a browser or server problem of many online services

by Vlad Turiceanu

Passionate about technology, Windows, and everything that has a power button, he spent most of his time developing new skills and learning more about the tech world. Coming… read more

Updated on March 17, 2023

Fact checked by
Alex Serban

After moving away from the corporate work-style, Alex has found rewards in a lifestyle of constant analysis, team coordination and pestering his colleagues. Holding an MCSA Windows Server… read more

Struggling with various browser issues? Try a better option: Opera OneYou deserve a better browser! Over 300 million people use Opera One daily, a fully-fledged navigation experience coming with various built-in packages, enhanced resource consumption, and great design. Here’s what Opera One can do:

• Optimize resource usage: Opera One uses your Ram more efficiently than Brave
• AI and User Friendly: New feature directly accessible from the sidebar
• No ads: Built-in Ad Blocker speeds up the loading of pages and protects against data-mining
• Gaming friendly: Opera GX is the first and best browser for gamers
• ⇒ Get Opera One

Many of our readers reported getting the no healthy upstream error message in a lot of instances.

Most of them are happening on their browser, trying to reach services like eBay, Spotify, Netflix and others.

However, it is also occurring in vCenter and VMWare, and that suggests the idea that this is a server problem.

Here, we’re going to provide solutions for the general problem but also target the issues in the most occurring situations and apps.

What is the meaning of no healthy upstream? 

The key word in this error message is upstream. And upstream in terms of coding means sending a pack of code to an admin to include in a certain app.

However, the no healthy upstream is totally different in the context of an error. It is simply a but that prevents running an app.

How can I fix the no healthy upstream error?

1. Clear the cache on your browser

1. Click the Menu button in Chrome (the three vertical dots) and select Settings.
2. Now, select Privacy & security from the left pane and click on Clear browsing data.
3. Mark only Cached images and files and click on Clear data.
4. Restart the browser and try visiting the service that returned the no healthy upstream error.

We have demonstrated this solution on Chrome since this error occurs in this browser most often. However, the process is very similar in other browsers as well.

2. Try a better browser

Since the no healthy upstream may be caused by the faulty communication between the browser and the server you are trying to reach, it’s only logical to look for alternatives.

We recommend using Opera because it’s a lot more stable and reliable than Chrome.

But the most important feature that sets it apart from Chrome is the presence of a free VPN service. You can also try to use it and see if it bypasses the no healthy upstream issue.

Besides that, you will benefit from a lot of native integrations like Facebook Messenger, WhatsApp, Instagram, and Telegram that will run seamlessly right from the browser.

They are readily optimized to work in Opera so this error shouldn’t appear.

Opera

With the integrated apps already optimized, you should have no issues browsing the web.

3. Restart your server and check the settings from your host

If no healthy upstream occurs on your server which is hosted by a third-party service, you should try restarting it.

Should the issue reappear even after the restart, check if the IP is provided from a DHCP service and if the DNS is correctly set up.

Communicate with your provider and get that information if you don’t know exactly what to do.

Read more about this topic

• Opera One Review: Futureproof Your PC With AI & Modularity
• Opera One vs Opera: Which One Should You Use & Why?
• Bing Homepage Quiz Not Working? Here’s How to Fix it
• Bad Request: Header Field Too Long [Error Fix]
• Microsoft Edge for Chromebook: How to Download & Install

No healthy upstream error in vCenter

There are many readers who encountered the vCenter no healthy upstream error. But before getting into other tweaking solutions, there are some preemptive procedures you can undergo to make it work:

➡ Stop vCenter and restart the service – It will take a little while to restart so wait a few minutes before trying to access it again on your browser

➡ Upgrade your virtual machine – The no healthy upstream error can be caused by a problem between vCenter and your VM so upgrading your machine should help

➡ Change the settings in vCenter – Access the VM Options button, go to General options and select VMWare Photon OS; one of our readers managed to fix the problem like that so it’s worth trying

➡ Choose an older hardware compatibility option – Use the Change Hardware Compatibility Wizard and choose a different configuration from the drop-down menu

By changing the setting your VM will be able to adapt to usage for older and more stable versions of the app and allow you to try out different compatibility possibilities.

No healthy upstream error on eBay

However, several of our readers have experienced the no healthy upstream error on eBay on their desktops and even their mobile apps. 

However, in this case, the no healthy upstream error is caused by technical issues on eBay and can only be fixed by them.

No healthy upstream error while using Netflix

If you’re using Netflix on your browser, the first two solutions from this guide should fix the problem.

To be more specific, try clearing your browser’s cache and cookies or, better yet, try another browser.

Another possible solution would be to restart your router. You should also try to open Netflix in an incognito window or use a VPN.

Opera has both functionalities so you can try the fixes with the same tool.

No healthy upstream error on Spotify

The situation on Spotify is exactly the same as for Netflix and it can be fixed by using the same methods we suggested above.

Clear the cache and cookies of your browser or use a better browser altogether. Opening Spotify in an incognito window and using a VPN are also to be considered.

As we recommended for Netflix, you could restart your router and see if that changes the situation.

What is an upstream timeout?

Upstream is a computer network term that refers to data transfer from a local computer or client to a remote host or server.

You get an upstream time out when the transfer takes too long, and the system interprets the request as failed.

As this issue is also related to virtual machines, you might be interested in our selection of the best VMs that work perfectly on Windows 11.

Here’s to hoping that you’ve found some enlightenment with this piece. Please share your thoughts in the comments section below.

Istio service mesh offers a multitude of solutions at network level 7 (L7) to define traffic routing, security, and application monitoring in a cloud environment. However, given the complexity of cloud-based networks, the host of devices involved, and the difficulty of visualizing effective changes made by Istio, it’s hard to debug the unpopular «no healthy upstream» error messages that often show up in Envoy logs.

This article attempts some pain relief in the form of quick guidance on how to respond to emergency calls demanding a resolution to «no healthy upstream» error messages and related errors such as «Applications in the Mesh are not available» or «Istio is broken.»

In my experience, 90% of these issues are caused by configuration problems in either the network or Istio. This article shows some troubleshooting tools you can use to identify such problems quickly, in the context of two recent cases that a Red Hat customer escalated to us.

It’s important to understand a few aspects of this customer’s architecture. The customer is running services in separate Red Hat OpenShift clusters, some of which are in the customer’s own on-premises infrastructure, while others span several countries in the EU region. Each OpenShift cluster has its own instance of a Red Hat OpenShift Service Mesh, Red Hat’s productized Istio service.

Kubernetes services make both intramesh and intermesh requests. But a service in this customer’s configuration always makes a local call. Integration and routing between services in the different clusters are performed by the mesh via a set of VirtualService, DestinationRule, and ServiceEntry resources that redirect the local call to a remote service.

A duplicate service

In our first real-life example, the customer complained that the service mesh somehow was causing cluster-to-cluster communications to fail, and reported the «no healthy upstream» message.

To identify a problem related to Istio configuration, I always use Istio’s Kiali console to visualize the network state and pinpoint where issues are occurring. Kiali allows you to «play back» network behavior, a nice feature that is very helpful if you’re dealing with a problem that is not occurring right now. Whether or not I discover the problematic service, I turn next to checking the logs of the Envoy proxy via either Kiali or OpenShift (using an oc logs <pod_name> -c istio-proxy commmand). The aim in both cases is to find the service for which the «no healthy upstream» error appears.

In this case, Kiali showed that 95% of the traffic to the destination service destination.mynamespace.svc.cluster.local was failing. My next resource was the istioctl command, which can provide a quick view of the state of the Envoy proxy and whether its configuration was updated correctly by Istio:

$ istioctl proxy-status
NAME CDS LDS EDS RDS PILOT VERSION
...
service-source-v1-74f955bd84-9lmnf.mynamespace SYNCED SYNCED SYNCED SYNCED istiod-86798869b8-bqw7c 1.5.0
...

Getting confirmation from the output that the mesh managed to keep all relevant service Envoy proxies up to date, I then checked the cluster names configured on the Envoy proxy of the client service pod. I focused only on the clusters related to the outbound service host for which logs showed the «no healthy upstream» message:

$ istioctl proxy-config cluster -i istio-system service-source-v1-74f955bd84- 9lmnf.mynamespace --fqdn service-destination.mynamespace.svc.cluster.local -o json | jq -r .[].name
outbound|80||service-destination.mynamespace.svc.cluster.local
inbound|80|9180-tcp|service-destination.mynamespace.svc.cluster.local   outbound|80|v1|service-destination.mynamespace.svc.cluster.local
outbound|80|v2|service-destination.mynamespace.svc.cluster.local

In the output, I noticed that the Istio configuration had defined two services (v1 and v2) for the cluster in question: outbound|80|v2|service-destination.mynamespace.svc.cluster.local. I then checked for the available endpoints for the v2 service:

$ istioctl proxy-config endpoints service-source-v1-74f955bd84-9lmnf.mynamespace --cluster "outbound|80|v2|service-destination.mynamespace.svc.cluster.local"   ENDPOINT STATUS OUTLIER CHECK CLUSTER
172.17.0.28:9180 HEALTHY OK outbound|80|v2|service destination.mynamespace.svc.cluster.local
172.17.0.29:9180 HEALTHY OK outbound|80|v2|service destination.mynamespace.svc.cluster.local

Then I proceeded to check the endpoints for v1. However, for the v1 service for outbound|80|v2|service destination.mynamespace, the mesh has no endpoints, and therefore no pods:

$ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development 2
cluster "outbound|80|v`|service-destination.mynamespace.svc.cluster.local"   ENDPOINT STATUS OUTLIER CHECK CLUSTER

This misconfiguration caused the «no healthy upstream» errors.

Checking the VirtualService for the destination, I noticed that 5% of the traffic is routed to v2, which agrees with what I saw also in Kiali, while 95% is routed to v1, which also explains why the customer saw 95% failures with the «no healthy upstream» message.

All the customer needed to do to fix the problem was to deploy service v1 or update the VirtualService to distribute all requests to v2.

Duplicate Envoy clusters

In a follow-up escalation, the «no healthy upstream» issue came up again. We followed the same troubleshooting approach as in the previous example, but in this case there was no VirtualService.

We saw multiple ServiceEntry definitions for multiple country destinations. We found it puzzling that all country destinations, apart from the one reported, had requests directed correctly. The following check verified that all clusters were reported as healthy except service-destination.remote-namespace.ocp4.customdomain.com:

$ oc exec istio-egressgateway-6567f7d756-4gvh8 -- curl localhost:15000/clusters |egrep 'health|remote-service-destination.remote-namespace.ocp4.customdomain.com'

In this case I looked for a log entry like:

outbound|443||remote-service-destination.remote namespace.ocp4.customdomain.com::10.128.2.21:443::health_flags::/failed_active_hc

Having verified the cluster as unhealthy, I turned to Istiod to ensure that no errors were being reported against this cluster. The Istiod pod reported:

2022-05-13T12:27:51.262316Z info ads Push finished: 5.709679819s {   "ProxyStatus": {
"pilot_duplicate_envoy_clusters": {
"outbound|443||remote-service-destination.remote-namespace.ocp4.customdomain.com": {
"proxy": "e2e-871-remote-namespace-c58f7f7f6-vljr6.e2e-871-remote namespace",
"message": "Duplicate cluster outbound|53||remote-service destination.remote-namespace.ocp4.customdomain.com found while pushing CDS"   }
},

This output indicates that Istiod was trying to apply a cluster configuration for which there was a duplicate. This information prompted me to check the applied ServiceEntry resources, which quickly revealed that there had been duplicate definitions for remote-service-destination.remote-namespace.ocp4.customdomain.com.

Istio administrative tools reveal the source of errors

A «no healthy upstream» error can be caused by Istio, a misconfigured network device, or actual network outages. Thus, it is difficult for the mesh operations team to pinpoint the cause or even predict its occurrence, because DevOps teams may unwittingly apply an incorrect configuration. This article provided guidance on how to establish the cause of such errors, determining whether they are or are not due to Istio configuration.

Even greater benefits can be realized when, as in the case of the customer in this example, the operations team applies observability monitoring and alerts against such occurrences, so that the team can be aware in advance of the issue and inform the relevant teams before an escalation occurs.

Last updated:
March 7, 2023

• #1

Добрый день! Не получается зайти на vcenter 7, в браузере появляется ошибка

Попробовал зайти через vcenter:5480 и там вижу ошибку:

File system /storage/log/ has run out of storage space.
Increase the size of disk /storage/log

Может в этом дело, подскажите как исправить ошибку ?

Последнее редактирование: 21.02.2023

• #2

Еще вижу ругань на недостаток свободного места:

Filesystem holding service vtsdb error logs is running low on available space.
Filesystem holding service vmware-vpostgres error logs is running low on available space.

root@localhost [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.3G 0 9.3G 0% /dev
tmpfs 9.4G 784K 9.4G 1% /dev/shm
tmpfs 9.4G 1.2M 9.4G 1% /run
tmpfs 9.4G 0 9.4G 0% /sys/fs/cgroup
/dev/sda3 46G 10G 34G 23% /
/dev/sda2 120M 27M 85M 25% /boot
tmpfs 9.4G 1.8M 9.4G 1% /tmp
/dev/mapper/core_vg-core 49G 53M 47G 1% /storage/core
/dev/mapper/log_vg-log 9.8G 9.7G 0 100% /storage/log
/dev/mapper/db_vg-db 9.8G 2.0G 7.4G 21% /storage/db
/dev/mapper/dblog_vg-dblog 15G 105M 14G 1% /storage/dblog
/dev/mapper/seat_vg-seat 25G 212M 24G 1% /storage/seat
/dev/mapper/netdump_vg-netdump 985M 2.5M 915M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 37M 9.3G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 37M 9.3G 1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr 98G 193M 93G 1% /storage/updatemgr
/dev/mapper/archive_vg-archive 49G 35G 12G 75% /storage/archive
/dev/mapper/vtsdb_vg-vtsdb 25G 76M 24G 1% /storage/vtsdb
/dev/mapper/vtsdblog_vg-vtsdblog 15G 73M 14G 1% /storage/vtsdblog
/dev/mapper/lifecycle_vg-lifecycle 98G 3.1G 90G 4% /storage/lifecycle

• #3

Расширьте диск вцентра с логами, в свойствах виртуалки

• #4

Расширьте диск вцентра с логами, в свойствах виртуалки

Проблема в том что он не дает расширить диск виртуальной машины

As this host’s resources are being managed by vCenter Server, some configuration options are disabled.
If you wish to change this VM’s resource settings, please do so from vCenter Server.

Я подключен к гипервизору через веб, а переконфигурировать виртуальную машину через vcenter не могу так как он не запаускается и пишет ошибку

Как теперь расширить диск ?

Lera

Активный участник

• #6

Подключитесь к vcenter через SSH и попробуйте запустить сервисы командой:
service-control --start --all
Статус можно посмотреть так
service-control --list

• #7

Подключитесь к vcenter через SSH и попробуйте запустить сервисы командой:
service-control --start --all
Статус можно посмотреть так
service-control --list

Пробую….

• #8

Не выполняется, ошибка вот такая

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, vpxd-svcs, sts services. Error: Operation timed out

• #9

Проверьте что у вас сертификаты не просрочены командами
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store TRUSTED_ROOTS —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store MACHINE_SSL_CERT —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store SMS —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store machine —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vpxd —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vpxd-extension —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vsphere-webclient —text | less

• #10

Проверьте что у вас сертификаты не просрочены командами
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store TRUSTED_ROOTS —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store MACHINE_SSL_CERT —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store SMS —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store machine —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vpxd —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vpxd-extension —text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list —store vsphere-webclient —text | less

Да, есть просроченные сертификаты. Но когда пытаюсь перегенерировать, то требуется пароль administrator@vsphere.local
Тот который ввожу не подходит. Но я точно успешно заходил с этими учетными данными в веб клиент vmware
Как сбросить пароль administrator@vsphere.local ?

• #11

Почему то сбросить пароль не получается
root@localhost [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcadmintool

==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

3
Please enter account UPN : cn=administrator,cn=users,dc=vSphere,dc=local
VmDirForceResetPassword failed (9106)

Lera

Активный участник

• #12

Может там не vsphere.local ? Проверьте
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

• #13

Может там не vsphere.local ? Проверьте
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

Да. С этим разобрался, но все равно сервисы не оживают.

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server — An error occurred when processing the metadata during vCenter Single Sign-On setup — Failed to connect to VMware Lookup Service https://vcsa:443/lookupservice/sdk — SSL certificate verification failed..

Lera

Активный участник

• #14

1. Log in to the vCenter Server via SSH/PuTTY session as root, and enable shell
2. Stop the vSphere client service using below command

service-control —stop vsphere-ui

1. Navigate to the vsphere-ui location to edit webclient.properties

cd /etc/vmware/vsphere-ui/

1. Before editing take a backupof webclient.properties using below command,
   • cp webclient.properties /var/tmp/webclient.properties.bak
2. Add the desired shortname under the sso.serviceprovider.alias.whitelist

• vi webclient.properties
• Type i to enter insert mode
• Remove the comment (#) for sso.serviceprovider.alias.whitelist=
• Add the shortname (comma separated if there are multiple values)
• Save and exit the VI editor by pressing Esc to exit insert mode, then wq!
• Example:
  • sso.serviceprovider.alias.whitelist=vcsa70

1. Start the vSphere client service.

service-control —start vsphere-ui

• #15

Да. С этим разобрался, но все равно сервисы не оживают.

Большая инфраструктура ? Может проще грохнуть этот и новый поставить ?

• #16

Большая инфраструктура ? Может проще грохнуть этот и новый поставить ?

не, всего 3 ESXi , distributed switch нету. Я уже тоже в эту сторону думаю

Lera

Активный участник

• #17

не, всего 3 ESXi , distributed switch нету. Я уже тоже в эту сторону думаю

тюю там делов на полчаса. Думаю так проще будет. Хотя интересно все же почему этот перестал работать

• #18

Поднял VCSA заново, все прицепил, добавил лицензии. Не получается теперь к домену AD подключить. Когда подключаю то получаю ошибку: idm client exception:error trying to join ad code 42500

• #19

Поднял VCSA заново, все прицепил, добавил лицензии. Не получается теперь к домену AD подключить. Когда подключаю то получаю ошибку: idm client exception:error trying to join ad code 42500

попробуй через CLI vcenter

/opt/likewise/bin/domainjoin-cli join <your domain> <user name>


• #20

попробуй через CLI vcenter

/opt/likewise/bin/domainjoin-cli join <your domain> <user name>


Пробовал, все равно ошибка

Error: ERROR_GEN_FAILURE [code 0x0000001f]