Create a crawler

POST 
/bv/aisk/v1/crawlers

This API is used to create a crawler resource.

Request

application/json

Body

required

crawler objectrequired

Required. The crawler resource to be created.

start_urls string[]

Input only. The list of URLs which the crawler will begin to crawl from.

Responses

200

A successful response.

application/json

Schema

Schema

crawler object

id string

Output only. The uuid of the crawler.

start_urls string[]

Input only. The list of URLs which the crawler will begin to crawl from.

status string

Possible values: [CRAWLER_STATUS_RUNNING, CRAWLER_STATUS_COMPLETED, CRAWLER_STATUS_FAILED]

Output only. The status of the crawler.

crawled_urls string[]

Output only. The list of URLs that the crawler found.

Example (from schema)

{
  "crawler": {
    "id": "string",
    "start_urls": [
      "string"
    ],
    "status": "CRAWLER_STATUS_RUNNING",
    "crawled_urls": [
      "string"
    ]
  }
}

400

A bad request response. The code is 3 means got an invalid argument. There are more HTTP status code mappings listed on here and gRPC code on here.

application/json

Schema

Schema

code int32
message string
details object[]
Array [
@type string
A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one "/" character. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).
In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:
• If no scheme is provided, https is assumed.
• An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
• Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)
Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. As of May 2023, there are no widely used type server implementations and no plans to implement one.
Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.
]

Example (from schema)

{
  "code": 0,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}

401

A unauthenticated response. The header authorization was missing or unidentified.

application/json

Schema

Schema

code int32
message string
details object[]
Array [
@type string
A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one "/" character. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).
In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:
• If no scheme is provided, https is assumed.
• An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
• Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)
Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. As of May 2023, there are no widely used type server implementations and no plans to implement one.
Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.
]

Example (from schema)

{
  "code": 0,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}

403

A forbidden response. It means that the provided authorization did not have enough permission to access the resource or the API.

application/json

Schema

Schema

code int32
message string
details object[]
Array [
@type string
A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one "/" character. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).
In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:
• If no scheme is provided, https is assumed.
• An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
• Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)
Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. As of May 2023, there are no widely used type server implementations and no plans to implement one.
Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.
]

Example (from schema)

{
  "code": 0,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}

500

A server error response. There are more HTTP status code mappings listed on here.

application/json

Schema

Schema

code int32
message string
details object[]
Array [
@type string
A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one "/" character. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).
In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:
• If no scheme is provided, https is assumed.
• An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
• Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)
Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. As of May 2023, there are no widely used type server implementations and no plans to implement one.
Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.
]

Example (from schema)

{
  "code": 0,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}

default

An unexpected error response.

application/json

Schema

Schema

code int32
message string
details object[]
Array [
@type string
A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one "/" character. The last segment of the URL's path must represent the fully qualified name of the type (as in path/google.protobuf.Duration). The name should be in a canonical form (e.g., leading "." is not accepted).
In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme http, https, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows:
• If no scheme is provided, https is assumed.
• An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error.
• Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.)
Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. As of May 2023, there are no widely used type server implementations and no plans to implement one.
Schemes other than http, https (or the empty scheme) might be used with implementation specific semantics.
]

Example (from schema)

{
  "code": 0,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}

Loading...