Skip to content

Improve personal memory: extraction, reconcile, procedures, hardening - #51

Open
Aayush Kataria (aayush3011) wants to merge 11 commits into
AzureCosmosDB:mainfrom
aayush3011:users/akataria/memoryImprovements
Open

Aayush Kataria (aayush3011) wants to merge 11 commits into
AzureCosmosDB:mainfrom
aayush3011:users/akataria/memoryImprovements

Conversation

@aayush3011

@aayush3011 Aayush Kataria (aayush3011) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Improves the personal memory pipeline: smarter extraction, duplicate folding in reconcile(), evidence-based procedural memory, safer writes and logs, and the matching Bicep changes. User-facing notes are in CHANGELOG.md.

What changed

  • Extraction: extract_memories-v3 is the default. It sees recent turns, the thread summary and the user's known facts as context, and facts can record assertion, subject, attribute, entities and source turns.
  • Reconcile: duplicate facts are folded and facts about one specific thing are combined before the contradiction pass. Passes accept a deadline, and the Function App stops them before the function timeout (ACTIVITY_TIME_BUDGET_SECONDS).
  • Procedural memory: learned incrementally from evidence as instructions, habits and skills. build_procedural_context() returns the standing rules.
  • Search: optional preference and procedure slots. Thread-scoped keyword and hybrid searches no longer send a partition key (FullTextScore() in ORDER BY clause does not take affect in Cosmos DB NoSQL full-text queries with partition key Azure/azure-sdk-for-python#42241).
  • Hardening: credentials are redacted on write, deleting a memory removes the records folded into it, logs and Function App failures no longer quote exception, query, memory or model text, and empty or refused model replies are handled.
  • Bicep: memories_summaries gets the vector and full-text index, and dedupPoolSize, activityTimeBudgetSeconds, chatCapacity and embeddingCapacity are new parameters.

Breaking changes

  • Procedural memory is reshaped, with no migration.
  • The log field exc_msg is now exc_detail, and a failed Function App activity reports only a short code.
  • An environment provisioned from an earlier template needs its memories_summaries container created again, because Cosmos DB sets a vector policy only at creation.

Validation

  • 2,524 unit tests pass; ruff check and ruff format --check are clean, and CI is green.
  • Run live against a Cosmos DB account and Azure OpenAI with Entra ID: 23 integration tests on the latest code (the TTL and change-feed tests earlier on the branch), all 18 samples and 4 notebooks earlier on the branch, a reconcile that combines facts and then corrects them, and a check that the v3 prompt cites real turn ids. Not run: a deployment of the Bicep template.
  • Still to check by hand: the thread-scoped keyword-search workaround against a real account, extraction quality with the v3 prompt on real conversations, and azd provision --preview against an environment created from the old template.
  • function_app/requirements.txt still pins SDK 0.3.0b2. Bump it with the SDK release before deploying the updated Function App code.

Extraction
- extract_memories-v3 is the default prompt. Extraction sees recent turns, the thread summary and the user's known facts as context.
- Facts can record assertion, subject, attribute, entities and source turns.

Reconcile
- reconcile() folds duplicate facts and combines facts about one specific thing before the contradiction pass.
- Passes accept a deadline, and the Function App stops them before the function timeout.

Procedural memory
- Procedures are learned incrementally from evidence as instructions, habits and skills (breaking reshape, no migration).

Search
- Optional preference and procedure slots in search_cosmos().
- Keyword and hybrid thread searches no longer send a partition key (Azure/azure-sdk-for-python#42241).

Hardening
- Credentials are redacted on write, and deleting a memory removes the records folded into it.
- Logs and Function App failures no longer quote exception, query, memory or model text.
- Empty and refused model replies are handled, plus smaller Function App config fixes.

Infra
- Bicep: the summaries container gets the vector and full-text index, and dedupPoolSize, activityTimeBudgetSeconds, chatCapacity and embeddingCapacity are new parameters.

See CHANGELOG.md for the breaking changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Reduce the extraction note to the new prompt and the variable that selects an earlier one.
- Shorten the procedural memory heading.
- Drop the supersedes note, the keyword-search bug-fix entry and the requirements pin entry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Tool-only replies with blank content are incorrectly retried as transient empty replies, causing persistent failures to defer extraction instead of being treated as non-retryable.

1 open issue
What changed in this PR

This PR improves the Agent Memory Toolkit’s personal-memory pipeline, from contextual extraction and reconciliation to evidence-based procedures, safer storage, and Function App processing.

Changes:

  • Adds contextual extraction, duplicate folding and combining, procedural learning, and optional search slots.
  • Hardens credential handling, logs, model-response handling, and time-budgeted processing.
  • Updates infrastructure, configuration, documentation, and regression coverage.
File Description
tests/​unit/​test_utils.py Tests preferred-region resolution.
tests/​unit/​test_thresholds.py Expands boolean-token coverage.
tests/​unit/​test_reconcile.py Adapts reconciliation tests to dedup counts.
tests/​unit/​test_query_builder.py Tests parameterized condition support.
tests/​unit/​test_procedural_synthesis.py Removes legacy procedural-synthesis tests.
tests/​unit/​test_no_exception_text_in_logs.py Adds source checks against exception-text logging.
tests/​unit/​test_logging.py Updates assertions for safe exception details.
tests/​unit/​test_log_describe.py Tests privacy-safe diagnostic descriptions.
tests/​unit/​test_fact_merge_fields.py Tests merge provenance and retirement reasons.
tests/​unit/​test_cosmos_memory_client.py Tests regions, counter expiry, and search defaults.
tests/​unit/​test_combine_thresholds.py Pins consolidation limits.
tests/​unit/​test_chat.py Tests empty replies, terminal failures, and safe logs.
tests/​unit/​store/​test_secret_redaction.py Tests redaction before embedding and persistence.
tests/​unit/​store/​test_search_routing.py Tests keyword and vector query routing.
tests/​unit/​store/​test_memory_store.py Updates thread-scoped search expectations.
tests/​unit/​services/​test_prompty_loader.py Tests prompt versions and contextual inputs.
tests/​unit/​services/​test_procedural_retrieval.py Updates retrieval tests for the procedure schema.
tests/​unit/​services/​test_pipeline_service.py Updates reconciliation fixtures and removes legacy procedure tests.
tests/​unit/​services/​test_parse_llm_json.py Checks parsing errors omit model output.
tests/​unit/​services/​test_extraction_batching.py Tests terminal and retryable reply classification.
tests/​unit/​services/​test_extract_prompt_selection.py Tests v3 defaults and earlier-prompt overrides.
tests/​unit/​services/​test_extract_dry.py Adapts extraction fixtures for context and retries.
tests/​unit/​services/​test_embedding_redaction.py Tests sanitized embedding inputs and summaries.
tests/​unit/​services/​test_absorbed.py Tests cascading deletion of folded records.
tests/​unit/​prompts/​test_memory_writing_prompts.py Checks secret-handling rules and prompt rendering.
tests/​unit/​prompts/​test_consolidation_prompts.py Tests consolidation prompt contracts.
tests/​unit/​processors/​test_inprocess.py Tests combined-fact reconciliation counts.
tests/​unit/​function_app/​test_synthesize_procedural_orchestrator.py Tests deadlines and reduced activity outputs.
tests/​unit/​function_app/​test_orchestrators.py Tests shared reconciliation deadlines.
tests/​unit/​function_app/​test_failure_logs.py Tests safe trigger and orchestration failures.
tests/​unit/​function_app/​test_config_time_budget.py Tests activity-budget defaults and bounds.
tests/​unit/​function_app/​test_config_bool_parity.py Checks Function App and SDK boolean parity.
tests/​unit/​function_app/​test_config_batch_size.py Tests rejection of empty batch windows.
tests/​unit/​aio/​test_reconcile_telemetry.py Updates async reconciliation result expectations.
tests/​unit/​aio/​test_procedural_synthesis.py Removes legacy async procedure tests.
tests/​unit/​aio/​test_cosmos_memory_client.py Tests async preferred regions and counter expiry.
tests/​unit/​aio/​test_chat.py Tests async reply handling and safe logs.
tests/​unit/​aio/​store/​test_secret_redaction.py Tests async write-path redaction.
tests/​unit/​aio/​store/​test_search_routing.py Tests async keyword and vector routing.
tests/​unit/​aio/​store/​test_memory_store.py Updates async thread-search expectations.
tests/​unit/​aio/​services/​test_procedural_retrieval_async.py Updates async procedure retrieval tests.
tests/​unit/​aio/​processors/​test_inprocess.py Tests async combined-fact counts.
Samples/​Scenarios/​scenario_tagging_and_filtering.py Formatting only.
Samples/​Scenarios/​scenario_remote_processor.py Formatting only.
Samples/​Scenarios/​scenario_remote_processor_async.py Formatting only.
Samples/​Scenarios/​scenario_rag_with_memory.py Formatting only.
Samples/​Scenarios/​scenario_multi_agent.py Formatting only.
Samples/​Scenarios/​scenario_memory_reconciliation.py Updates reconciliation guidance and cascade-aware cleanup.
Samples/​Scenarios/​scenario_customer_support.py Formatting only.
Samples/​Scenarios/​scenario_counter_tuning.py Formatting only.
Samples/​Scenarios/​scenario_chat_memory.py Formatting only.
Samples/​Quickstarts/​quickstart_local.py Formatting only.
Samples/​Processing/​processing_thread_summary.py Formatting only.
Samples/​Notebooks/​Demo_function_app.ipynb Notebook formatting only.
Samples/​Notebooks/​Demo_function_app_async.ipynb Notebook formatting only.
Samples/​Advanced/​advanced_search_patterns.py Formatting only.
infra/​README.md Documents indexes, capacity, and processing controls.
infra/​modules/​functions.bicep Passes reconciliation and time-budget settings.
infra/​modules/​cosmos.bicep Adds summary vector and full-text indexes.
infra/​main.parameters.json Binds new deployment environment variables.
infra/​main.bicep Exposes processing budgets and model capacity.
function_app/​triggers/​change_feed.py Sanitizes trigger logs and propagated failures.
function_app/​shared/​error_redaction.py Adds safe activity-failure wrappers.
function_app/​shared/​counters.py Replaces exception text with safe diagnostics.
function_app/​shared/​config.py Shares boolean parsing and validates processing settings.
function_app/​orchestrators/​user_summary.py Wraps summary activities with error redaction.
function_app/​orchestrators/​thread_summary.py Wraps thread activities with error redaction.
function_app/​orchestrators/​synthesize_procedural.py Adds deadlines and reduced result payloads.
function_app/​orchestrators/​extract_memories.py Adds safe failures and reconciliation deadlines.
function_app/​orchestrators/​extract_episodes.py Wraps episode extraction with error redaction.
function_app/​local.settings.json.template Adds reconciliation pool and budget settings.
Docs/​troubleshooting.md Explains search filtering, safe failures, and budgets.
Docs/​operations.md Documents expiry, deletion, and pipeline controls.
Docs/​local_testing.md Documents default counter expiry.
Docs/​azure_testing.md Updates procedure and reconciliation validation guidance.
Docs/​architecture/​orchestrators.md Documents processing passes, cursors, and redaction boundaries.
CHANGELOG.md Summarizes features, fixes, and breaking changes.
azure/​cosmos/​agent_memory/​thresholds.py Defines context, consolidation, and counter-expiry limits.
azure/​cosmos/​agent_memory/​store/​_search_helpers.py Adds result fields, procedure filters, and keyword routing.
azure/​cosmos/​agent_memory/​services/​_absorbed.py Implements cascading deletion of folded records.
azure/​cosmos/​agent_memory/​prompts/​user_summary.prompty Adds secret-exclusion instructions.
azure/​cosmos/​agent_memory/​prompts/​user_summary_update.prompty Forbids retaining secrets in updated profiles.
azure/​cosmos/​agent_memory/​prompts/​synthesize_procedural.prompty Removes the legacy synthesis prompt.
azure/​cosmos/​agent_memory/​prompts/​summarize.prompty Adds secret-exclusion instructions.
azure/​cosmos/​agent_memory/​prompts/​summarize_update.prompty Forbids retaining secrets in updated summaries.
azure/​cosmos/​agent_memory/​prompts/​extract_memories.prompty Adds credential-exclusion instructions.
azure/​cosmos/​agent_memory/​prompts/​extract_memories-v2.prompty Adds credential-exclusion instructions.
azure/​cosmos/​agent_memory/​prompts/​extract_episode.prompty Prioritizes secret exclusion over exact capture.
azure/​cosmos/​agent_memory/​prompts/​dedup.prompty Corrects the explanation of duplicate-pass ordering.
azure/​cosmos/​agent_memory/​prompts/​dedup_verify.prompty Adds conservative duplicate and containment checks.
azure/​cosmos/​agent_memory/​prompts/​combine_memories.prompty Adds detail-preserving memory combination.
azure/​cosmos/​agent_memory/​processors/​inprocess.py Includes combined facts in reconciliation counts.
azure/​cosmos/​agent_memory/​chat.py Adds empty-reply retries and safe failure logging.
azure/​cosmos/​agent_memory/​auto_trigger.py Sanitizes automatic-processing failure reports.
azure/​cosmos/​agent_memory/​aio/​processors/​inprocess.py Mirrors combined-fact counting asynchronously.
azure/​cosmos/​agent_memory/​aio/​chat.py Mirrors reply handling and safe logging asynchronously.
azure/​cosmos/​agent_memory/​aio/​auto_trigger.py Sanitizes async automatic-processing failures.
azure/​cosmos/​agent_memory/​_utils.py Resolves ordered Cosmos read regions.
azure/​cosmos/​agent_memory/​_slots.py Plans and fills typed search-result slots.
azure/​cosmos/​agent_memory/​_query_builder.py Supports prebuilt parameterized conditions.
azure/​cosmos/​agent_memory/​_counters.py Sanitizes failure-stamping diagnostics.
azure/​cosmos/​agent_memory/​_base/​base_client.py Stores resolved preferred-region configuration.
.env.template Documents region and extraction-prompt overrides.

Review effort: Balanced

Comment thread azure/cosmos/agent_memory/chat.py Resolved
A reply that finished with tool calls but had empty or whitespace-only content was retried as a transient empty reply. It now raises unexpected_tool_response, like a reply with no content, so extraction does not keep retrying it. This applies to the sync and async chat clients.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants