# Operational Readiness Checklist & Known Caveats > A consolidated, actionable production-readiness checklist across every dimension, the upgrade strategy for Meshery, and the known caveats and limitations to plan around. --- LLMS index: [llms.txt](/llms.txt) --- This page brings the whole set together: a single checklist you can work through before declaring a Meshery deployment production-ready, the upgrade strategy to adopt, and the known caveats to design around. Each item links back to the page where it is explained in full. ## Upgrade strategy Meshery is a composition of components - Server, UI, Operator, MeshSync, Broker, Adapters, and `mesheryctl` - some of which upgrade together and some independently. For Kubernetes production deployments: - **Use Helm and pin versions.** Upgrade with `helm upgrade`, and pin the image to an **immutable version tag** rather than tracking `stable-latest`, so upgrades are deliberate and reproducible ([security hardening](/installation/production/security-hardening/)). - **Use upgrade-friendly probe values.** Capability reloading can make the Server briefly unavailable during an upgrade. The chart provides `values-upgrade.yaml` (startup probe, higher failure thresholds) so pods are not killed mid-initialization: ```bash helm upgrade meshery meshery/meshery --namespace meshery \ -f https://raw.githubusercontent.com/meshery/meshery/master/install/kubernetes/helm/meshery/values-upgrade.yaml \ --wait --timeout 10m ``` For reproducible production upgrades, download `values-upgrade.yaml`, version-control it alongside your own values, and reference your local copy (with a pinned chart and image version) rather than fetching from `master` at upgrade time. - **Choose a release channel deliberately.** `stable` for production; `edge` only for testing. The channel is reflected by `RELEASE_CHANNEL`. - **Mind component groupings.** See the [Upgrade Guide](/installation/upgrades/) for which components move together (e.g., Server/UI/Load Generators/Database) versus independently (Adapters, `mesheryctl`). The Operator and its controllers effectively move **with the Server**: the chart version a Server deploys to managed clusters tracks its own release, and the operator image is pinned inside that chart. - **Expect a published chart, not necessarily the matching one.** Before installing, the Server validates the version it wants against the chart repository's published index, so it never hands Helm a chart that does not exist. A version that is not published yet resolves to the newest published chart, and a version below the minimum deployable chart is raised to the oldest published chart at or above it; release candidates are never chosen automatically. Every substitution is reported in the events feed. Budget outbound access from Meshery Server to the chart repository accordingly ([egress requirements](/installation/production/networking-and-connectivity/#egress-requirements)). - **Let the Server own the Operator.** On Server-managed clusters, do not hand-upgrade or hand-configure the `meshery-operator` Helm release - the Server's reconciliation re-applies the chart version it resolves and reverts manual changes. Upgrading the Server is the supported way to upgrade the Operator; the chart's CRD update Job refreshes the `Broker`/`MeshSync` CRD schemas on each upgrade (Helm alone never updates CRDs). When you must hold a cluster at a specific operator chart, set `operator.version` on that connection - an explicit pin is honored and never substituted, and is refused with a visible error if it names an unpublished or moving version. See [How Meshery Server manages Meshery Operator](/installation/upgrades/#how-meshery-server-manages-meshery-operator). - **Expect CRDs to persist.** The `brokers.meshery.io` and `meshsyncs.meshery.io` CRDs (and their objects) deliberately survive operator uninstalls and Helm release deletion; include the explicit `kubectl delete crd ...` step in decommissioning runbooks only when permanent removal is intended. - **Rehearse and roll back.** Because durable state lives with the Remote Provider and the local database is a cache, rolling back the deployment is low-risk for data - validate the rollback path anyway. On rollback, managed clusters converge back to the operator chart the older Server resolves, automatically. The minimum deployable chart version guards that derived default, but it is not a blanket guarantee: an explicit `operator.version` is honored as written even when it names a chart below the minimum, and if the repository publishes nothing at or above the minimum the newest published chart is deployed with a warning that the Operator may not become ready. - **Edge caches need no purge.** If a CDN or caching reverse proxy fronts Meshery, its UI cache busts itself on upgrade - hashed asset URLs change and the HTML `ETag` follows the build/release version - so a manual cache purge is not required in your upgrade pipeline ([networking](/installation/production/networking-and-connectivity/)). ## Known caveats and limitations Plan around these known characteristics - they are by design and shape good production architecture: - **The database is an ephemeral, single-writer cache.** It is SQLite/Bitcask on local disk, not a shared multi-writer datastore. Durable state must come from a [Remote Provider](/installation/production/authentication-and-identity/); this also shapes how Server replication behaves ([HA & Resiliency](/installation/production/high-availability-and-resiliency/)). - **The Broker persists messages in memory only.** No persistent volume is used. Brief interruptions are bridged by NATS topic persistence, but a far-behind or unavailable Server consumer lets Broker memory climb ([sizing](/installation/production/infrastructure-sizing-and-performance/)). - **The Local Provider has no authentication.** Never run shared production on it; preselect a Remote Provider ([identity](/installation/production/authentication-and-identity/)). - **WebSocket support is mandatory at the ingress.** Without it the UI loads but never updates live ([networking](/installation/production/networking-and-connectivity/)). - **OAuth requires the external callback URL.** Behind a proxy, an unset or wrong `MESHERY_SERVER_CALLBACK_URL` breaks login. - **Out-of-cluster requires a reachable Broker endpoint.** Cross-cloud reachability of the Broker is the most common multi-cloud failure ([multi-cloud](/installation/production/multi-cluster-and-multi-cloud/)). - **Discovery cost scales with cluster size and churn.** Without MeshSync blacklisting, very large/churny clusters dominate Server memory and database growth. - **Policy evaluation is time-boxed.** Large designs may hit `POLICY_EVAL_TIMEOUT` (default `3m`); tune CPU and the timeout together. - **The chart ships permissive defaults for portability.** No resource requests/limits, empty security contexts, and a `stable-latest`, `Always`-pull image are convenient for getting started but are **not** production settings - set them explicitly. ## The consolidated production-readiness checklist Work through these before going live. Each group links to its source page. ### Deployment model - [ ] Topology chosen deliberately (in-cluster vs. out-of-cluster; Docker vs. Kubernetes), with Kubernetes + Helm for production. [Deployment Models](/installation/production/deployment-models/) - [ ] One Operator/MeshSync/Broker per managed cluster (or embedded mode chosen deliberately). ### Sizing & performance - [ ] Explicit CPU/memory **requests and limits** set for Server, Operator, MeshSync, and Broker. [Sizing](/installation/production/infrastructure-sizing-and-performance/) - [ ] Server memory headroom for the MeshSync snapshot; Broker memory headroom for bursts. - [ ] MeshSync `informer_config` blacklist tuned for large clusters. - [ ] Storage provisioned for the Server cache; autoscaling configured where appropriate. ### High availability & resiliency - [ ] Liveness/readiness probes enabled and tuned. [HA & Resiliency](/installation/production/high-availability-and-resiliency/) - [ ] Anti-affinity spreads replicas across nodes/zones. - [ ] Helm values, env config, and connection definitions in version control (your real backup). - [ ] Redeploy-from-config recovery drill validated. ### Networking & connectivity - [ ] Ingress fronts Meshery with TLS; **WebSocket upgrades** verified. [Networking](/installation/production/networking-and-connectivity/) - [ ] `MESHERY_SERVER_CALLBACK_URL` set to the external URL. - [ ] Broker endpoint reachable from the Server and restricted to its origin. - [ ] Egress to the Remote Provider and registries allowed; network policies applied. - [ ] Any fronting CDN/caching proxy honors origin cache headers (immutable assets, `no-cache` HTML) and never caches `/api/*` or the WebSocket. ### Security & identity - [ ] **Remote Provider preselected** (`PROVIDER`, `PROVIDER_BASE_URLS`); Local Provider not used. [Identity](/installation/production/authentication-and-identity/) - [ ] Least-privilege RBAC; `rbac.nodes` only where required; per-cluster credentials. - [ ] Hardened `securityContext` (non-root, dropped caps, read-only root FS with writable data volume) on Server and Operator. [Security Hardening](/installation/production/security-hardening/) - [ ] Secrets (kubeconfig, provider, pull) sourced from Secrets/external manager. - [ ] Images pinned to immutable tags from a trusted/mirrored registry. ### Extensions - [ ] Every enabled extension attributed to a publisher you trust, and reviewed before it reached production. [Trusting an extension](/installation/production/security-hardening/#trusting-an-extension) - [ ] Unused extension points left disabled; adapters enabled only where needed (`ADAPTER_URLS`, chart adapter subcharts off by default). - [ ] Adapters given scoped ServiceAccounts (`serviceAccountNameOverride`) rather than sharing `meshery-server`, and reachable only from Meshery Server. - [ ] Extension packages pinned (`SKIP_DOWNLOAD_EXTENSIONS`), and the capability set pinned (`PROVIDER_CAPABILITIES_FILEPATH`) where it should not drift. [Providers reference](/reference/extensibility/providers/#runtime-configuration-options) - [ ] Extension versions validated against the Meshery version you run. [Extension compatibility](/reference/extensibility/verify-compatibility/) - [ ] Egress policy covers every destination an enabled extension requires. - [ ] Rollback path known: removing a server-side extension requires a Meshery Server restart, not just a configuration change. ### Multi-cluster & multi-cloud - [ ] One least-privilege kubeconfig context per cluster. [Multi-Cloud](/installation/production/multi-cluster-and-multi-cloud/) - [ ] Per-cloud node-watch RBAC and Broker endpoint reachability validated. - [ ] Private connectivity for private clusters; cross-region latency assessed. ### Monitoring & observability - [ ] External uptime check on `/healthz/ready`; workload metrics scraped. [Monitoring & KPIs](/installation/production/monitoring-observability-and-kpis/) - [ ] Dedicated alerts on Server memory and Broker memory; per-cluster connection health and provider reachability monitored. - [ ] OpenTelemetry tracing to a real collector; logs centralized at `LOG_LEVEL=4`. - [ ] `mesheryctl system check` scheduled as a synthetic check. ### Operations - [ ] Upgrade procedure uses `values-upgrade.yaml`; `stable` release channel. [Upgrade Guide](/installation/upgrades/) - [ ] Runbooks reference the [troubleshooting guides](/guides/troubleshooting/meshery-operator-meshsync/). - [ ] Ownership, on-call, and escalation defined for the Meshery deployment. ## Where to go next - Revisit any area above via its linked page in this set. - For component internals, see the [Meshery Architecture](/concepts/architecture/) reference. - For runtime configuration, see the [Server Environment Variables](/installation/advanced/environment-variables/) reference.

Recent Discussions with "meshery" Tag

View all discussions tagged with meshery