核验学习与规划会议前身卷并补齐发现范围 #40
@@ -62,6 +62,8 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
|
||||
|
||||
早期主系列的Workshop名称按[ICML/IROS精确届次证据](docs/coverage/historical-primary-series-2026-11-09.md)识别,保留原名与范围证据;代码支持不代表已补齐生产数据。
|
||||
|
||||
EWSL、EPS、ECP、EWSP 的[具体前身卷与完整快照要求](docs/coverage/historical-predecessors-2026-11-09.md)单独核验,保留原始系列及事件/出版年份;旧候选文件缺少 ECP/EWSP 时会明确提示,避免静默遗漏。
|
||||
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)目前核验了 2012 版式,可从缓存或已校验快照采集标题、作者及页面提供的摘要;发表形式单独保存,全文/扩展摘要中未明确的投稿 track 保持待明确,不根据会话主题推定主会轨道。
|
||||
|
||||
`paper-atlas collect --help` 查看来源;`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# EWSL 与规划会议前身的来源范围
|
||||
|
||||
本项扩展 DBLP 元数据适配:核对具体历史卷,保留 EWSL、EPS、ECP、EWSP 原始系列名称,并使用已有会议别名关系关联 ECML-PKDD 或 ICAPS。它不把历史论文的原始会议名称替换成后来的会议,也不从卷的存在推断有逐篇数据。
|
||||
|
||||
## 证据与已核验边界
|
||||
|
||||
[ECML-PKDD 2011 官方历史页](https://ecmlpkdd2011.org/pastConferences.php)与 [ICAPS 官方系列历史](https://www.icaps-conference.org/conference-series/)分别提供系列前身关系。2026-11-09 冻结页面 SHA-256 为:
|
||||
|
||||
- ECML-PKDD:`c3ecdb032aeacf6a365aace5457773827d1405872ad9bc6be6fc05851d5e68d6`。
|
||||
- ICAPS:`de1d217660cb819825f58112cdaf774d0f68da6ed87e3e90a59055e52f152081`。
|
||||
|
||||
与固定 DBLP 2026-11-01 快照的完整 proceedings 目录对照后,仅对下列具体卷识别前身主系列:
|
||||
|
||||
| 原始系列 | 事件年 | DBLP 卷 key | 卷出版年 | 原始逐篇书目计数 |
|
||||
|---|---:|---|---:|---:|
|
||||
| EWSL | 1987 | `conf/ecml/1987` | 1987 | 16 |
|
||||
| EWSL | 1988 | `conf/ecml/1988` | 1988 | 21 |
|
||||
| EWSL | 1989 | `conf/ecml/1989` | 1989 | 0 |
|
||||
| EWSL | 1991 | `conf/ecml/1991` | 1991 | 44 |
|
||||
| EPS | 1990 | `conf/aips/1990` | 1990 | 48 |
|
||||
| ECP | 1997 | `conf/ecp/1997` | 1997 | 35 |
|
||||
| ECP | 1999 | `conf/ecp/1999` | 2000 | 28 |
|
||||
| EWSP | 1991 | `conf/ewsp/1991` | 1991 | 10 |
|
||||
|
||||
合计 202 条是原始 `inproceedings` 书目计数,尚未经过前言等材料排除、身份核查与生产验收,不能作为新论文数量。EWSL 1989 在当前 catalogue 中有卷信息,但未匹配到 `inproceedings` 条目;其他记录类型和其他来源仍待核查。代码支持它的范围规则不表示已补上这一年的论文。
|
||||
|
||||
核验同时要求来源 family、booktitle 基名、空轨道后缀、完整 proceedings key、完整标题及卷出版年与固定证据一致。邻近年份、标题附加内容、`w` 卷后缀或工作坊 booktitle 均不能继承此范围。ECP、EWSP 及 EPS 的未知卷保持待审。原有 AIPS、ICML/IROS、AISTATS/COLT 规则保持各自边界。
|
||||
|
||||
EWSL 的主轨道仍属于 `EWSL`,通过 `ECML-PKDD` 别名纳入家族查询;EPS、ECP、EWSP 保留原名,通过 `ICAPS` 别名关联。`main` 在此表示该原始系列的主轨道。原始 booktitle、作者、论文书目年、卷标题、卷出版年、事件年及历史页 URL 分开保存在 metadata 中。例如 ECP 1999 的事件年 1999 与卷出版年 2000 都保留,不用其中一个覆盖另一个。
|
||||
|
||||
## 候选文件与完整快照
|
||||
|
||||
旧的窄范围 `dblp-ai-candidates.jsonl` 未纳入 `ecp`、`ewsp` family,但完整 catalogue 的 proceedings 与 volume_counts 中有它们。新的 catalogue 导出会纳入这两个 family。
|
||||
|
||||
选择 ICAPS 且截止年达到 EWSP 1991 或 ECP 1997 时,若候选 manifest 缺少相应 family,`collect dblp --candidates` 会明确报错,要求使用原始完整 `--archive`。错误发生在写输出之前,不会把遗漏当作成功采集。选择其他会议或截止 EPS 1990 的旧候选采集不受这项新家族检查影响。
|
||||
|
||||
在已确认的远程运行环境中使用完整快照:
|
||||
|
||||
```sh
|
||||
paper-atlas collect dblp \
|
||||
--catalogue /data/paper-index/collected/dblp/dblp-catalogue.sqlite \
|
||||
--archive /path/to/pinned-dblp.xml.gz --dtd /path/to/pinned-dblp.dtd \
|
||||
--venue ECML-PKDD --venue ICAPS --through-year 2001 \
|
||||
--out /data/paper-index/collected/historical-predecessors
|
||||
```
|
||||
|
||||
完整快照与 DTD 均沿用现有固定身份校验。该命令也会输出已覆盖的其他年度;生产导入前须单独筛选目标卷与年度,检查 raw source、前言材料、稳定标识、标题、作者及现有投稿,不能直接导入整个输出目录。不会下载论文 PDF。
|
||||
|
||||
## 验证和当前限制
|
||||
|
||||
回归检查使用独立保存的八卷原始目录 fixture,合成论文只用于隔离测试;不代表真实逐篇来源。检查原始系列/别名、完整标题和年份边界、旧候选文件显式失败、扩展 family 的 catalogue 导出、原生导入后的元数据与关系。
|
||||
|
||||
代码、测试、采集和生产导入是不同阶段。新范围不表示本批数据已采集或已导入,更不表示公开拒稿、摘要或最终去重完整。原始书目仅提供发表条目证据,不是原始决定信;没有记录不能推断为拒稿或未举办。后续各批需保留独立验收报告,保护既有跨会投稿、决定与标签历史。
|
||||
@@ -20,7 +20,7 @@ DTD_MD5='96e4283e71886dec2173718957293648'
|
||||
DTD_NAMES={'dblp.dtd','dblp-2023-06-28.dtd'}
|
||||
RELEASE='https://doi.org/10.4230/dblp.xml.2026-11-01'
|
||||
# Broad discovery candidates, including historical names; not a venue mapping.
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps kr atal aamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps ecp ewsp kr atal aamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
|
||||
|
||||
def records(archive,dtd_bytes):
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Exact predecessor volumes, with original names and independent family histories."""
|
||||
ECML_HISTORY = 'https://ecmlpkdd2011.org/pastConferences.php'
|
||||
ICAPS_HISTORY = 'https://www.icaps-conference.org/conference-series/'
|
||||
|
||||
# family/year: booktitle, original series, target family, publication year, full title
|
||||
VOLUMES = {
|
||||
('ecml', 1987): ('EWSL', 'EWSL', 'ECML-PKDD', 1987,
|
||||
'Progress in Machine Learning - Proceedings of EWSL 87: 2nd European Working Session on Learning, Bled, Yugoslavia, May 1987'),
|
||||
('ecml', 1988): ('EWSL', 'EWSL', 'ECML-PKDD', 1988,
|
||||
'Proceedings of the Third European Working Session on Learning, EWSL 1988, Turing Institute, Glasgow, UK, October 3-5, 1988'),
|
||||
('ecml', 1989): ('EWSL', 'EWSL', 'ECML-PKDD', 1989,
|
||||
'Proceedings of the 4th European Working Session on Learning, Montpellier, France, 4-6 December 1989'),
|
||||
('ecml', 1991): ('EWSL', 'EWSL', 'ECML-PKDD', 1991,
|
||||
'Machine Learning - EWSL-91, European Working Session on Learning, Porto, Portugal, March 6-8, 1991, Proceedings'),
|
||||
('aips', 1990): ('Expert Planning Systems', 'EPS', 'ICAPS', 1990,
|
||||
'Proceedings of the First International Conference on Expert Planning Systems, Brighton, UK, June 27-29, 1990'),
|
||||
('ecp', 1997): ('ECP', 'ECP', 'ICAPS', 1997,
|
||||
"Recent Advances in AI Planning, 4th European Conference on Planning, ECP'97, Toulouse, France, September 24-26, 1997, Proceedings"),
|
||||
('ecp', 1999): ('ECP', 'ECP', 'ICAPS', 2000,
|
||||
"Recent Advances in AI Planning, 5th European Conference on Planning, ECP'99, Durham, UK, September 8-10, 1999, Proceedings"),
|
||||
('ewsp', 1991): ('EWSP', 'EWSP', 'ICAPS', 1991,
|
||||
"European Workshop on Planning, EWSP '91, Sankt Augustin, Germany, March 18-19, 1991, Proceedings"),
|
||||
}
|
||||
|
||||
|
||||
def predecessor_scope(family, base, suffix, volume, year, title):
|
||||
definition = VOLUMES.get((family, year))
|
||||
if definition is None:
|
||||
return None
|
||||
book, original, target, publication_year, expected_title = definition
|
||||
if (suffix or base != book or volume['key'] != f'conf/{family}/{year}'
|
||||
or title != expected_title or volume['fields'].get('year') != [str(publication_year)]):
|
||||
return None
|
||||
return {'original_series': original, 'target_family': target,
|
||||
'history_url': ECML_HISTORY if target == 'ECML-PKDD' else ICAPS_HISTORY,
|
||||
'proceedings_key': volume['key'], 'proceedings_title': title,
|
||||
'event_year': year, 'proceedings_year_fields': list(volume['fields']['year'])}
|
||||
|
||||
|
||||
def missing_candidate_families(targets, families, through_year):
|
||||
# These two families were absent from the earlier narrowed candidate JSONL,
|
||||
# although their proceedings and volume_counts exist in the full catalogue.
|
||||
required = {family for (family, year), definition in VOLUMES.items()
|
||||
if family in ('ecp', 'ewsp') and definition[2] in targets and year <= through_year}
|
||||
return sorted(family for family in required if not families.get(family))
|
||||
@@ -15,6 +15,7 @@ from urllib.parse import parse_qs,unquote,urlsplit
|
||||
from paper_atlas.config import DEFAULT_REGISTRY
|
||||
from paper_atlas.collectors.dblp import XML_SHA,DTD_MD5,RELEASE,records
|
||||
from paper_atlas.collectors.dblp_scope import scope
|
||||
from paper_atlas.collectors.dblp_predecessors import missing_candidate_families
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.snapshots import digest_file
|
||||
from paper_atlas.io.publication_ids import mlsys_id
|
||||
@@ -88,6 +89,8 @@ def normalize_row(row,volumes,targets,include_workshops=False,through_year=2026)
|
||||
'scope_rule':edition['rule'],'author_names_raw':fields.get('author',[]),'publtype':row.get('publtype'),
|
||||
'bibliography':{key:value for key,value in fields.items() if key in ('booktitle','year','ee','url','pages','volume','number','publisher','series','isbn')},
|
||||
'ambiguous_identifiers':ambiguous,'publication_evidence':'inproceedings bibliography; not an original decision note'}
|
||||
if edition.get('historical_predecessor_evidence'):
|
||||
metadata['historical_predecessor_evidence'] = edition['historical_predecessor_evidence']
|
||||
if edition.get('joint_edition_evidence'):
|
||||
metadata['joint_edition_evidence'] = edition['joint_edition_evidence']
|
||||
if edition.get('track_scope_note'):
|
||||
@@ -124,8 +127,9 @@ def collect(catalogue,out,venues=None,candidates=None,archive=None,dtd=None,incl
|
||||
families = manifest.get('candidate_family_counts', {})
|
||||
missing = [venue for venue, family in [('MM', 'mm'), ('MLSys', 'mlsys')]
|
||||
if venue in targets and not families.get(family)]
|
||||
missing += ['ICAPS/'+family for family in missing_candidate_families(targets, families, through_year)]
|
||||
if missing:
|
||||
raise ValueError('Candidate file predates requested families ' + ', '.join(missing) + '; use the original --archive')
|
||||
raise ValueError('Candidate file omits requested families ' + ', '.join(missing) + '; use the original --archive')
|
||||
out=Path(out);out.mkdir(parents=True,exist_ok=True)
|
||||
with (out/'dblp.normalize.lock').open('a') as lock:
|
||||
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Explicit bibliographic aliases; a directory family alone is not venue evidence."""
|
||||
import re
|
||||
|
||||
from paper_atlas.collectors.dblp_predecessors import predecessor_scope
|
||||
|
||||
|
||||
# Ordered long/joint names precede their shorter forms.
|
||||
BOOKS = {
|
||||
@@ -27,7 +29,8 @@ BOOKS = {
|
||||
'pkdd':[('ECML/PKDD','ECML-PKDD',[]),('ECML PKDD','ECML-PKDD',[]),('PKDD','PKDD',['ECML-PKDD'])],
|
||||
'icra':[('ICRA','ICRA',[])], 'iros':[('IROS','IROS',[])],
|
||||
'rss':[('Robotics: Science and Systems','RSS',[])], 'corl':[('CoRL','CoRL',[])],
|
||||
'aips':[('ICAPS','ICAPS',[]),('AIPS','AIPS',['ICAPS'])], 'icaps':[('ICAPS','ICAPS',[])],
|
||||
'aips':[('ICAPS','ICAPS',[]),('AIPS','AIPS',['ICAPS']),('Expert Planning Systems','EPS',['ICAPS'])], 'icaps':[('ICAPS','ICAPS',[])],
|
||||
'ecp':[('ECP','ECP',['ICAPS'])], 'ewsp':[('EWSP','EWSP',['ICAPS'])],
|
||||
'kr':[('KR','KR',[])], 'atal':[('AAMAS','AAMAS',[])],
|
||||
'icassp':[('ICASSP','ICASSP',[])],
|
||||
'interspeech':[('INTERSPEECH','INTERSPEECH',[]),('EUROSPEECH','EUROSPEECH',['INTERSPEECH']),('ICSLP','ICSLP',['INTERSPEECH']),('ECST','ECST',['INTERSPEECH'])],
|
||||
@@ -135,7 +138,11 @@ def scope(row,volume,through_year=2026):
|
||||
if len(bibliographic)!=1 or not re.fullmatch(r'\d{4}',bibliographic[0]):return None,'ambiguous_bibliographic_year'
|
||||
if int(bibliographic[0])!=year and not re.search(r'\b'+str(year)+r'\b',title):return None,'unverified_year_difference'
|
||||
text=(suffix+' '+title).casefold();track='main'
|
||||
historical = historical_track(family, base, suffix, volume['key'], year, title)
|
||||
predecessor = predecessor_scope(family, base, suffix, volume, year, title)
|
||||
if (family in ('ecp', 'ewsp') or (family == 'aips' and base == 'Expert Planning Systems')) and not predecessor:
|
||||
return None, 'unverified_historical_predecessor'
|
||||
historical = (('main', 'Historical predecessor; exact volume, full title, event and publication years verified; original series retained.', predecessor['history_url'])
|
||||
if predecessor else historical_track(family, base, suffix, volume['key'], year, title))
|
||||
if historical:track=historical[0]
|
||||
elif 'findings' in text:track='findings'
|
||||
elif 'shared task' in text:track='shared_task'
|
||||
@@ -165,6 +172,8 @@ def scope(row,volume,through_year=2026):
|
||||
joint_evidence = 'https://www.eurai.org/ecai/history'
|
||||
result = {'venue':venue,'venue_aliases':aliases,'year':year,'track':track,'bibliographic_year':int(bibliographic[0]),
|
||||
'rule':family+':'+base,'proceedings_title':title,'edition_year_basis':year_basis}
|
||||
if predecessor:
|
||||
result['historical_predecessor_evidence'] = predecessor
|
||||
if joint_evidence:
|
||||
result['joint_edition_evidence'] = joint_evidence
|
||||
if historical:
|
||||
|
||||
+314
@@ -0,0 +1,314 @@
|
||||
{
|
||||
"source_snapshot_sha256": "0376ad019b216553faff0a14e1757b2d4c7a3b84c1beacf0511ef83695211b18",
|
||||
"scope": "Pinned proceedings metadata, not authentic example paper records or evidence that all volumes contain indexed papers. Test papers are synthetic.",
|
||||
"volumes": [
|
||||
{
|
||||
"key": "conf/ecml/1987",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Ivan Bratko",
|
||||
"Nada Lavrac"
|
||||
],
|
||||
"title": [
|
||||
"Progress in Machine Learning - Proceedings of EWSL 87: 2nd European Working Session on Learning, Bled, Yugoslavia, May 1987"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Sigma Press, Wilmslow"
|
||||
],
|
||||
"year": [
|
||||
"1987"
|
||||
],
|
||||
"isbn": [
|
||||
"1-85058-088-X"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl87.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1987",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 16
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1988",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Derek H. Sleeman"
|
||||
],
|
||||
"title": [
|
||||
"Proceedings of the Third European Working Session on Learning, EWSL 1988, Turing Institute, Glasgow, UK, October 3-5, 1988"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Pitman Publishing"
|
||||
],
|
||||
"year": [
|
||||
"1988"
|
||||
],
|
||||
"isbn": [
|
||||
"0-273-08800-9"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl88.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1988",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 21
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1989",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Katharina Morik"
|
||||
],
|
||||
"title": [
|
||||
"Proceedings of the 4th European Working Session on Learning, Montpellier, France, 4-6 December 1989"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Pitman and Morgan Kaufman Publishers"
|
||||
],
|
||||
"year": [
|
||||
"1989"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1991",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Yves Kodratoff"
|
||||
],
|
||||
"title": [
|
||||
"Machine Learning - EWSL-91, European Working Session on Learning, Porto, Portugal, March 6-8, 1991, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"482"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1991"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-53816-X"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/BFb0016999"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl91.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1991",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 44
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecp/1997",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Sam Steel",
|
||||
"Rachid Alami 0001"
|
||||
],
|
||||
"title": [
|
||||
"Recent Advances in AI Planning, 4th European Conference on Planning, ECP'97, Toulouse, France, September 24-26, 1997, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"ECP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"1348"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1997"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-63912-8"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/3-540-63912-8"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecp/ecp1997.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1997",
|
||||
"booktitle": "ECP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 35
|
||||
}
|
||||
],
|
||||
"expected_original_series": "ECP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecp/1999",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Susanne Biundo",
|
||||
"Maria Fox 0001"
|
||||
],
|
||||
"title": [
|
||||
"Recent Advances in AI Planning, 5th European Conference on Planning, ECP'99, Durham, UK, September 8-10, 1999, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"ECP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"1809"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"2000"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-67866-2"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/10720246"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecp/ecp1999.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1999",
|
||||
"booktitle": "ECP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 28
|
||||
}
|
||||
],
|
||||
"expected_original_series": "ECP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/ewsp/1991",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Joachim Hertzberg"
|
||||
],
|
||||
"title": [
|
||||
"European Workshop on Planning, EWSP '91, Sankt Augustin, Germany, March 18-19, 1991, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"522"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1991"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-54364-3"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/BFb0052944"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ewsp/ewsp1991.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1991",
|
||||
"booktitle": "EWSP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 10
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/aips/1990",
|
||||
"fields": {
|
||||
"title": [
|
||||
"Proceedings of the First International Conference on Expert Planning Systems, Brighton, UK, June 27-29, 1990"
|
||||
],
|
||||
"booktitle": [
|
||||
"Expert Planning Systems"
|
||||
],
|
||||
"publisher": [
|
||||
"AAAI"
|
||||
],
|
||||
"year": [
|
||||
"1990"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/aips/aips1990.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1990",
|
||||
"booktitle": "Expert Planning Systems",
|
||||
"record_type": "inproceedings",
|
||||
"count": 48
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EPS",
|
||||
"expected_target_family": "ICAPS"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Original series, exact edition boundaries, and complete candidate discovery."""
|
||||
from contextlib import closing
|
||||
from copy import deepcopy
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors.dblp import catalogue, XML_SHA
|
||||
from paper_atlas.collectors.dblp_records import collect, normalize_row
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURE = json.loads((Path(__file__).parent/'fixtures/historical_predecessors.json').read_text())
|
||||
EXAMPLES = {row['key']: row for row in FIXTURE['volumes']}
|
||||
|
||||
|
||||
def example(key, *, bibliographic_year=None):
|
||||
fixed = EXAMPLES[key]
|
||||
family = key.split('/')[1]
|
||||
year = int(key.split('/')[-1])
|
||||
volume = {'record_type': 'proceedings', 'key': key, 'fields': deepcopy(fixed['fields'])}
|
||||
row = {'record_type': 'inproceedings', 'key': f'conf/{family}/SyntheticExample{year}', 'publtype': None,
|
||||
'fields': {'title': [f'Synthetic historical research example {family} {year}'],
|
||||
'author': ['Jane Doe 0001', 'John Smith'], 'booktitle': fixed['fields']['booktitle'],
|
||||
'year': [str(bibliographic_year or year)], 'crossref': [key],
|
||||
'ee': [f'https://doi.org/10.1234/synthetic.{family}.{year}']}}
|
||||
return row, volume
|
||||
|
||||
|
||||
class HistoricalPredecessorTests(unittest.TestCase):
|
||||
def normalize(self, key, **kwargs):
|
||||
row, volume = example(key, **kwargs)
|
||||
fixed = EXAMPLES[key]
|
||||
result, reason = normalize_row(row, {key: volume}, {fixed['expected_target_family']})
|
||||
self.assertIsNone(reason)
|
||||
self.assertIsNotNone(result)
|
||||
return result
|
||||
|
||||
def test_ewsl_primary_series_retains_original_name_and_family_alias(self):
|
||||
for year in [1987, 1988, 1989, 1991]:
|
||||
with self.subTest(year=year):
|
||||
record = self.normalize(f'conf/ecml/{year}')
|
||||
self.assertEqual((record['venue'],record['venue_aliases'],record['year'],record['track']), ('EWSL',['ECML-PKDD'],year,'main'))
|
||||
proof = record['metadata']['historical_predecessor_evidence']
|
||||
self.assertEqual(proof['original_series'], 'EWSL')
|
||||
self.assertEqual(proof['history_url'], 'https://ecmlpkdd2011.org/pastConferences.php')
|
||||
self.assertEqual(record['metadata']['bibliography']['booktitle'], ['EWSL'])
|
||||
self.assertEqual(record['metadata']['author_names_raw'], ['Jane Doe 0001','John Smith'])
|
||||
self.assertEqual(record['authors'], ['Jane Doe','John Smith'])
|
||||
self.assertEqual(EXAMPLES['conf/ecml/1989']['bibliographic_counts'], [])
|
||||
|
||||
def test_eps_ecp_ewsp_keep_original_series_instead_of_renaming_to_icaps(self):
|
||||
for key in ['conf/aips/1990','conf/ecp/1997','conf/ecp/1999','conf/ewsp/1991']:
|
||||
with self.subTest(key=key):
|
||||
record = self.normalize(key)
|
||||
fixed = EXAMPLES[key]
|
||||
self.assertEqual(record['venue'], fixed['expected_original_series'])
|
||||
self.assertEqual(record['venue_aliases'], ['ICAPS'])
|
||||
self.assertEqual(record['track'], 'main')
|
||||
proof = record['metadata']['historical_predecessor_evidence']
|
||||
self.assertEqual(proof['history_url'], 'https://www.icaps-conference.org/conference-series/')
|
||||
self.assertEqual(proof['proceedings_title'], fixed['fields']['title'][0])
|
||||
self.assertEqual(proof['proceedings_year_fields'], fixed['fields']['year'])
|
||||
|
||||
def test_altered_titles_keys_suffixes_publication_years_and_unreviewed_editions_stay_held(self):
|
||||
for key in ['conf/ecml/1988','conf/aips/1990','conf/ecp/1997','conf/ewsp/1991']:
|
||||
row, volume = example(key)
|
||||
for changed in ['title','key','suffix','publication_year']:
|
||||
with self.subTest(key=key, changed=changed):
|
||||
r, v = deepcopy(row), deepcopy(volume)
|
||||
if changed == 'title': v['fields']['title'][0] += ', Affiliated Workshop'
|
||||
elif changed == 'key': v['key'] += 'w'; r['fields']['crossref'] = [v['key']]
|
||||
elif changed == 'suffix': r['fields']['booktitle'][0] += ' (Workshop)'
|
||||
else: v['fields']['year'] = ['2001']
|
||||
result, _ = normalize_row(r, {v['key']: v}, {EXAMPLES[key]['expected_target_family']})
|
||||
self.assertIsNone(result)
|
||||
row, volume = example('conf/ecp/1997')
|
||||
row['fields']['year'] = ['2001']; row['fields']['crossref'] = ['conf/ecp/2001']
|
||||
volume['key'] = 'conf/ecp/2001'; volume['fields']['year'] = ['2001']
|
||||
self.assertIsNone(normalize_row(row, {volume['key']:volume}, {'ICAPS'})[0])
|
||||
|
||||
def test_ecp1999_event_and_bibliographic_years_are_independently_retained(self):
|
||||
record = self.normalize('conf/ecp/1999', bibliographic_year=2000)
|
||||
self.assertEqual(record['year'], 1999)
|
||||
self.assertEqual(record['metadata']['bibliographic_year'], 2000)
|
||||
self.assertEqual(record['metadata']['bibliography']['year'], ['2000'])
|
||||
self.assertEqual(record['metadata']['historical_predecessor_evidence']['proceedings_year_fields'], ['2000'])
|
||||
self.assertEqual(record['metadata']['historical_predecessor_evidence']['event_year'], 1999)
|
||||
|
||||
def test_old_candidates_require_new_families_only_when_requested_years_need_them(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary); row, volume = example('conf/aips/1990')
|
||||
db = root/'dblp-catalogue.sqlite'
|
||||
with closing(sqlite3.connect(db)) as connection, connection:
|
||||
connection.execute('CREATE TABLE proceedings(key TEXT PRIMARY KEY,metadata_json TEXT)')
|
||||
connection.execute('INSERT INTO proceedings VALUES(?,?)', (volume['key'],json.dumps(volume)))
|
||||
candidate = root/'candidate.jsonl'; candidate.write_text(json.dumps(row)+'\n')
|
||||
db.with_name('dblp-ai-candidates.manifest.json').write_text(json.dumps({
|
||||
'snapshot_sha256': XML_SHA, 'candidate_sha256': hashlib.sha256(candidate.read_bytes()).hexdigest(),
|
||||
'candidate_records': 1, 'candidate_family_counts': {'aips':1},
|
||||
'record_types': {'proceedings':1,'inproceedings':1}}))
|
||||
# Tiny CI fixtures may live on tmpfs; production reserve is tested separately.
|
||||
with patch('paper_atlas.collectors.dblp_records.shutil.disk_usage', return_value=SimpleNamespace(free=16*1024**3)):
|
||||
result = collect(db, root/'early', ['ICAPS'], candidates=candidate, through_year=1990)
|
||||
self.assertEqual((result['normalized_records'],result['files'][0]['venue']), (1,'EPS'))
|
||||
for year, missing in [(1991,'ICAPS/ewsp'),(1997,'ICAPS/ecp')]:
|
||||
out = root/str(year); out.mkdir(); sentinel=out/'previous.txt';sentinel.write_text('accepted previous output')
|
||||
with self.subTest(year=year), self.assertRaisesRegex(ValueError, missing):
|
||||
collect(db, out, ['ICAPS'], candidates=candidate, through_year=year)
|
||||
self.assertEqual(sentinel.read_text(), 'accepted previous output')
|
||||
self.assertFalse((out/'dblp-normalization.json').exists())
|
||||
|
||||
def test_new_catalogue_includes_previously_missing_families_without_decision_inference(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary);dtd=root/'dblp.dtd';dtd.write_bytes(b'<!ENTITY eacute "é">')
|
||||
archive=root/'dblp.xml.gz'
|
||||
body=''.join(f'<inproceedings key="conf/{family}/Synthetic"><title>Synthetic example</title><year>{year}</year><booktitle>{book}</booktitle><crossref>conf/{family}/{year}</crossref></inproceedings>' for family,year,book in [('ecp',1997,'ECP'),('ewsp',1991,'EWSP')])
|
||||
with gzip.open(archive,'wb') as stream:stream.write(('<?xml version="1.0"?><!DOCTYPE dblp SYSTEM "dblp.dtd"><dblp>'+body+'</dblp>').encode())
|
||||
result=catalogue(archive,dtd,root/'out',hashlib.sha256(archive.read_bytes()).hexdigest(),hashlib.md5(dtd.read_bytes()).hexdigest())
|
||||
rows=[json.loads(line)for line in Path(result['file']).read_text().splitlines()]
|
||||
self.assertEqual({r['key'] for r in rows}, {'conf/ecp/Synthetic','conf/ewsp/Synthetic'})
|
||||
self.assertEqual(result['candidate_records'], 2)
|
||||
self.assertTrue(all('status' not in r and 'venue' not in r for r in rows))
|
||||
|
||||
def test_native_import_keeps_original_series_aliases_and_full_source_evidence(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root=Path(temporary);db=root/'sample.sqlite'
|
||||
for key in ['conf/ecml/1988','conf/ecp/1999']:
|
||||
record=self.normalize(key);path=root/(key.replace('/','-')+'.jsonl')
|
||||
path.write_text(json.dumps(record,ensure_ascii=False)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','venue':record['venue'],'year':record['year'],'track':'main','count':1,'list_url':'https://doi.org/10.4230/dblp.xml.2026-11-01','file_sha256':hashlib.sha256(path.read_bytes()).hexdigest(),'status':'partial','missing_abstract_count':1}))
|
||||
ingest_file(db,path)
|
||||
with closing(sqlite3.connect(db)) as connection:
|
||||
rows=connection.execute('SELECT v.acronym,a.alias,e.year FROM editions e JOIN venues v ON v.id=e.venue_id JOIN venue_aliases a ON a.venue_id=v.id ORDER BY v.acronym').fetchall()
|
||||
self.assertEqual(rows,[('ECP','ICAPS',1999),('EWSL','ECML-PKDD',1988)])
|
||||
metadata=[json.loads(r[0])for r in connection.execute('SELECT metadata_json FROM source_records')]
|
||||
self.assertEqual({r['historical_predecessor_evidence']['original_series'] for r in metadata}, {'ECP','EWSL'})
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decisions WHERE status=?',('accepted',)).fetchone()[0],2)
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(),[])
|
||||
|
||||
def test_existing_aips_normalization_remains_original_aips(self):
|
||||
row, volume=example('conf/aips/1990');row['fields']['booktitle']=['AIPS'];row['fields']['year']=['1994'];row['fields']['crossref']=['conf/aips/1994']
|
||||
volume['key']='conf/aips/1994';volume['fields']['title']=['AIPS 1994 Proceedings'];volume['fields']['year']=['1994']
|
||||
record,reason=normalize_row(row,{volume['key']:volume},{'ICAPS'})
|
||||
self.assertIsNone(reason);self.assertEqual((record['venue'],record['venue_aliases']),('AIPS',['ICAPS']))
|
||||
self.assertNotIn('historical_predecessor_evidence',record['metadata'])
|
||||
Reference in New Issue
Block a user