http: improve 404 Not Found response message

When a URL path is not found, return a non-empty message with the 404 status
code to help the user understand what went wrong. If the URL path was not
prefixed with '/v1/', suggest that may be the cause of the problem (which is a
common mistake).

.changelog/11818.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+http: when a URL path is not found, include a message with the 404 status code to help the user understand why (e.g., HTTP API endpoint path not prefixed with /v1/)
+```

agent/http.go

@@ -587,6 +587,12 @@ func (s *HTTPHandlers) Index(resp http.ResponseWriter, req *http.Request) {
 	// Check if this is a non-index path
 	if req.URL.Path != "/" {
 		resp.WriteHeader(http.StatusNotFound)
+
+		if strings.Contains(req.URL.Path, "/v1/") {
+			fmt.Fprintln(resp, "Invalid URL path: not a recognized HTTP API endpoint")
+		} else {
+			fmt.Fprintln(resp, "Invalid URL path: if attempting to use the HTTP API, ensure the path starts with '/v1/'")
+		}
 		return
 	}