Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In the middleware doStream, in the incoming params, maxRetries is default value #3577

Open
adolphnov opened this issue Nov 8, 2024 · 2 comments
Labels

Comments

@adolphnov
Copy link

adolphnov commented Nov 8, 2024

Description

In the middleware doStream, the obtained maxRetries is incorrect. The parameter obtained from prepareCallSettings does not include maxRetries:

fn: async (doStreamSpan3) => ({
              startTimestampMs: now2(),
              // get before the call
              doStreamSpan: doStreamSpan3,
              result: await model.doStream({
                mode,
                ...prepareCallSettings(settings),
                inputFormat: promptFormat,
                prompt: promptMessages,
                providerMetadata,
                abortSignal,
                headers
              })
            })

Check the streamText definition, settings do not carry user-defined maxRetries:

async function streamText({
  model,
  tools,
  toolChoice,
  system,
  prompt,
  messages,
  maxRetries,
  abortSignal,
  headers,
  maxToolRoundtrips = 0,
  maxSteps = maxToolRoundtrips != null ? maxToolRoundtrips + 1 : 1,
  experimental_continueSteps: continueSteps = false,
  experimental_telemetry: telemetry,
  experimental_providerMetadata: providerMetadata,
  experimental_toolCallStreaming: toolCallStreaming = false,
  experimental_activeTools: activeTools,
  onChunk,
  onFinish,
  onStepFinish,
  _internal: {
    now: now2 = now,
    generateId: generateId3 = originalGenerateId4,
    currentDate = () => /* @__PURE__ */ new Date()
  } = {},
  ...settings
}) {

Code example

return {
        wrapGenerate: async ({ doGenerate, params, model }) => {
            log.info('doGenerate called');
            log.debug(`params: ${JSON.stringify(params, null, 2)}`);
            log.info(`provider: ${model.provider}, modelId: ${model.modelId} `);
            const logs = getLogSingleton(config);
            activeTools.length > 0 ? logs.tool.model = model.modelId : logs.chat.model.push(model.modelId);
            const result = await doGenerate();
            log.info(`generated text: ${result.text}`);
            return result;
        },
...
}

Additional context

No response

@adolphnov adolphnov changed the title In the middleware dostream, the obtained maxTries is incorrect. In the middleware doStream, in the incoming params, maxTries is default value Nov 8, 2024
@adolphnov adolphnov changed the title In the middleware doStream, in the incoming params, maxTries is default value In the middleware doStream, in the incoming params, maxRetries is default value Nov 8, 2024
@lgrammel
Copy link
Collaborator

maxRetries should not be exposed to the middleware. It is handled on the generateText / streamText level.

@adolphnov
Copy link
Author

Got it. It is suggested to modify the signature here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants